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ABSTRACT 


This thesis describes a model diagnostic problem and a computer 
program designed to deal with this problem. The model diagnostic 
problem is an abstract problem, A major contention of this thesis, 
however, is that this problem subsumes the principal features of a 
number of ostensibly different real diagnostic problems including 
certain problems of medical diagnosis and the diagnosis of machine 
failures, A second major contention of this thesis is that strate- 
gies for the solution of the model diagnostic problem can be formu- 
lated in terms sufficiently explicit to permit their incorporation 
in a computer program, 


The model diagnostic problem assumes that the system being di- 
agnosed (e.g, a person, or.machine) is in one of a finite number of 
known states, Tests can be performed at some cost to discover attrib- 
utes of the system, for example signs or symptoms in medical diagno- 
sis, The current state of the system is to be deduced from the ob- 
served attributes and past experience with similar systems, In the 
model, this experience is represented principally in terms of proba- 
bilities (e.g. the conditional probability of a certain attribute 
given the system state), 


The statement of the model diagnostic problem requires that the 
diagnostician also account for the cost of various misdiagnoses, In 
particular for each pair of states i and j, the cost of misdiagnosing 
state j as state i, 1,,, is given. Thus the diagnostician must bal- 
ance the cost of performing additional tests against the expected 
reduction in the cost of misdiagnosis, This requirement suggests the 


value of sequential diagnosis. 


ii 


A computer program was developed to solve the model diagnostic 
problem, It consists of 1) an inference function which is based on 
a Bayesian analysis of attributes and includes a flexible way of 
dealing with non-independent attributes, 2) a pattern-sorting function 
which allows the program to detect irrelevant attributes and patterns 
of attributes corresponding to two different system states, and 3) 
a test selection function which employs various heuristics to select 
good tests for the user of the program to perform on the system under 
consideration, The diagnostic program is specialized for a particular 
problem by providing it with the appropriate experience. The pro- 
gram is embedded in an environment (set of programs) which facili- 
tates the study of various diagnostic strategies. 


The diagnostic program was implemented on the time-sharing 
system at Project MAC, It was applied to two medical problems, the 
diagnosis of congenital heart disease, and the diagnosis of primary 
bone tumors, The results obtained here suggest 1) that a computer 
program can be of considerable value as a diagnostic tool, and 2) 
that it is quite advantageous for such a program to perform sequential 
diagnosis as it interacts with the user, 
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Chapter 1 


DIAGNOSTIC PROBLEMS AND PROCESSES 


There are many problem areas in which attention is focused 
on some system, In these areas, the principal problem is to ascer- 
tain the current state of the given system, In general terms such 
a problem is a diagnostic problem. The problem-solver or diagnosti- 
cian is equipped for his task with information distilled from past 
experience with such systems, and he attempts to couple this gen- 
eral knowledge with specific observations or tests of the given 
system in such a way that he can deduce the identity of the current 
state, The extent of the general knowledge, its organization, 
and the particular manner in which it is brought to bear on the di- 
agnostic problem, the diagnostic process, may vary considerably 
among different problem areas, but the general nature of the prob- 
lem persists, 

Thus the medical diagnostician deals with the problem of dis- 
covering the "state" of the patient. Through training and experi- 
ence, the physician has learned the sign and symptom patterns asso- 
ciated with possible diseases from which the patient can suffer, One 
problem is the effective utilization of this experience which is 
framed in terms of the abstraction of the disease and the reality of 


the individual patient. An additional complication arises from the 


fact that different diseases may result in similar signs and symp- 
toms, The physician exploits his general knowledge or experience 

in the selection of a sequence of tests to apply to the patient. 

The results of these tests provide him with information from which 
he constructs a more complete picture of the health of the patient, 
These tests may include simple questions as in the history-taking 

or complicated medical procedures such as in an exploratory opera- 
tion, Since tests may exact a high cost (in terms of risk to the pa- 
tient, patient discomfort, the time of skilled persons, money, etc.), 
it is the additional task of the diagnostician to properly balance 
this cost against the potential usefulness of the test results. For 
these and other reasons, medical diagnosis is often a complex and 
difficult intellectual problem, 

A second example of a diagnostic problem is that of debugging 
computer programs. A program containing one or more errors can be 
thought of as a system for which it is desired to determine the state, 
The state in this case is characterized by the particular combination 
of errors, The programmer brings his past experience with a variety 
of programs to bear on this diagnostic problem, By controlling the 
inputs to the program, applying traces, or altering instruction se- 
quences, or employing a post mortem, he can perform a range of tests 
on the program, The results of these tests may suggest new tests as 
well as providing the programmer with new insight into the problem 


currently confronting him, Like medical diagnosis, program debugging 


is often a difficult task, requiring considerable judgment both in 
the selection of tests and the interpretation of results. 

The research reported here is concerned with a particular diag- 
nostic problem and a diagnostic process for solving that problem, It 
has several aims, The first is to formulate the model of the diagnos- 
tic problem in such a way that the definition subsumes the principal 
features of problems in a number of ostensibly different problem 
areas, For example, the definition might apply both to medical diag- 
nosis and to program debugging, although it might not be the particu- 
lar definition employed by diagnosticians in the respective areas. 
That such a model can be formulated is the major contention of this 
thesis. The second aim is to develop and investigate strategies for 
the solution of this model diagnostic problem, Because they are to 
be stated in terms of an abstract problem, such strategies will be 
independent of any real diagnostic problem, These diagnostic pro- 
cedures then are to be embodied in a computer program, This step 
serves two purposes, First, the program provides an explicit state- 
ment of the diagnostic strategies, and thus facilitates the testing 
of these strategies on particular problems, Second, if the strate- 
gies in the program prove effective in practical applications, the 
program could be of considerable value in computer-aided diagnosis, 
In the event that this approach were successful, the resulting pro- 
gram may be useful in a number of distinct diagnostic problems, since 


the methods it employed would be problem-independent, The second 
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major contention of this thesis is that given a model for the diag- 
nostic problem, effective strategies for the solution of the problem 
can be formulated in terms appropriate for their implementation in a 
program, 

Such a program for diagnosis could be embedded in an environment 
(other programs) which would permit two different uses of the program. 
First, the program could be applied to actual diagnostic problems so 
that its effectiveness could be determined. Second, the environment 
could permit the study of a variety of artificial problems, each 
designed to test a particular aspect of program performance, The first 
type of application might be termed "open diagnosis"; and the second, 
"closed diagnosis," Closed diagnosis may facilitate the development 
of improved diagnostic strategies, 

In order for a diagnostic problem to exist, one must have at 
least some knowledge of the nature of the system being considered, 
Further the various states of the system must manifest themselves 
through certain observable attributes. | It should also be possible 
to apply tests to the system at some cost to obtain more attributes, 
Finally, the general knowledge of the system must include some com- 
prehension of the relationships among signs, states, and tests, The 
prerequisites are satisfied by the two examples of diagnostic prob- 


lems presented above, In fact, in simplest terms, this is the basis 


lthe term attribute is used in this thesis to denote any observable 
manifestation of system state which is employed in the deductive phase 
of diagnosis, For example, it includes both signs and symptoms in 
medicine, 


for the diagnostic problem studied in this work, 


A Brief Outline of a Diagnostic Process 


The basic outline of a diagnostic process is as follows, Be- 
cause the observation of certain initial attributes suggest a diag- 
nostic problem in some system, the diagnostician wishes to ascertain 
the current state of the system, He selects a test (based on some 
criterion) and applies it to the system, The application of the test 
yields to update his current view of the problem, He then applies 
another test and obtains more attributes, This process continues un- 
til the diagnostician makes a decision about the current state, Now 
this is a most sketchy outline of the diagnostic process, There can 
be a great deal of sophisticated information processing during each 
iteration of the process, The point is that test selection and in- 
ference are the two principal features of diagnosis as performed in 
a number of distinct areas, The outline above seems equally appli- 
cable to medical diagnosis, qualitative chemical analysis, and the 
problem of diagnosing a malfunctioning automobile, At this level, then, 
the diagnostic processes in these and other areas exhibit considerable 
similarity. Inference and test selection appear to be the keys to 
diagnostic strategies of some generality. If it could be demon- 
strated that these features of the process necessarily differ funda- 
mentally from area to area, than there would be little hope for the 
formulation of general diagnostic strategies, In fact, as will be 


shown in this work, there is reason to believe quite the contrary. 


It appears that, for a number of areas, problem-independent diagnostic 
strategies can be developed, Note that the strategies employed by 
experts in different fields may be quite dissimilar, there is no re- 
quirement that the strategies developed here resemble theirs. The 
‘eriterion by which strategies will be judged is how effective they 
are in particular applications, not how closely they approximate those 
currently used by human experts, 

The diagnostic process then merits careful study for several 
reasons, First, as indicated above, variations of this. problem arise 
in many different contexts and so the problem is of general interest, 
Second, the nature of the diagnostic problem is such that it often 
requires a great deal of intellectual effort to solve it, and any 
means of improving the problem-solving process will be of consider- 
able value, Finally, the general form of the problem suggests the 
value of a man-machine partnership in the problem-solving process, 
Before such a partnership can be established, however, the diagnostic 
process must be carefully explored in order to determine respective 


parts to be played by man and machine, 


Some Further Comments on the Difficulties of Diagnosis 


Diagnostic problems on the whole are difficult ones, particularly 
for non-experts, Moreover, a great many diagnostic problems consti- 
tute considerable challenges to the skill of even the most expert 
diagnostician, Several factors contribute to the complexity of the 


diagnostic problem, First, an expert diagnostician must be aware of 


a large number of relationships among system states and attributes, 
As evidence of this, consider the considerable training required to 
develop the skills of a medical diagnostician, Observation of many 
different attributes may be required to identify a particular state, 
and a given attribute may suggest many possible states, These facts 
coupled with the often large number of states and attributes require 
the diagnostician to master considerable amounts of information, 

Often the relationships mentioned above are known only in proba- 
bilistic terms, In such a case, the task of the diagnostician is 
complicated by the need for some form of probability analysis, a 
task which generally proves quite difficult for human beings, The 
accurate assessment of probabilities for a large number of possible 
states given observed attributes requires extensive training and ex- 
perience, 

Another factor complicating the task of the diagnostician is 
the difficulty of establishing and maintaining an appropriate struc- 
‘ture for all the information relevant to the diagnostic area, Much 
of the usefulness of that information in the diagnostic process ac- 
crues from its organization, A major portion of the expert's skill 
is derived from his ability to associate particular attributes or at- 
tribute patterns with possible system states and subsequent testing 
strategies, Again extensive experience and training are required to 
organize the relevant information into a useful associative structure, 


Unfortunately such a structure is not easily maintained, Associations 
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which are seldom used may be effectively lost to the diagnostician, 
As a result, his field of competence tends to become narrow, This 
tendency is accelerated when the diagnostician must devote considerable 
effort to the mastering of a continual stream of newly-relevant in- 
i formation, 
A computer program to provide general diagnostic assistance to 
tes user would help circumvent some of these difficulties, One of 
the significant advantages to be gained from the use of a computer 
is the sheer bulk of information which it can maintain, A diagnos- 
tic program would be able to deal with extremely large information 
structures, Since the program would be independent of the content 
of the information structure which it employed, that content could 
be continually updated without affecting the operation of the pro- 
gram (although better information should result in better program 
performance). 

The amount of logical and probabilistic inference with which 
the program could cope would exceed that comprehensible to a human 
being, This capability would permit the more extensive exploration 
of possible testing strategies, Because the program could consider 
more possible diagnoses than a human bean: it would provide a strong 
safeguard that a particular state is not overlooked in the diagnosis, 
Finally, a diagnostic program which was "table-driven" would be of 
all the more value because of its potential applicability to a 


variety of problems, . 


Note that diagnostic strategies suited for a computer are not 
necessarily suited for a human diagnostician, While human diagnos- 
ticians possess many special skills and hence serve as good sources 
of information about diagnosis, the purpose of this research does 
not restrict the set of possible strategies to those employed by 
humans, The goal is to develop strategies which enable the pecu- 
liar capabilities of the computer to be exploited, Additional in- 
sight into the nature of the human diagnostic process and the dis- 
covery of ways to improve it would be a valuable, but derivative 


result of this research. 


A Preface to the Material Which Follows 

This thesis describes a computer program for diagnosis and 
presents the results of some experiments performed with this pro- 
gram. The design of the program was strongly influenced by the model 
diagnostic problem chosen for this research, Although later chapters 
contain detailed discussions of this problem, a brief summary of its 
principal characteristics is presented here to provide some perspec- 
tive on the problem, 

The statement of the diagnostic problem considered here assumes 
that the system is in one of a finite number of states, The object 
of the diagnosis is to identify the current state of the system, Ex- 
perience with similar systems is assumed to be available, This ex- 


perience is in the form of probabilities for the various states and 
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probabilities of attributes given state, Test costs are constant 
and known, Furthermore the application of a test does not change 
the state of the system, Tests are also assumed to be accurate, 
Finally, it is assumed that the decision loss for each possible mis- 
diagnosis is given in the same units as test costs, This work, 
then, is concerned with the development of strategies to solve 
diagnostic problems which can be stated in keeping with these as- 
sumptions, 

Chapter 2 examines some of the research reported in the lit- 
erature which has direct relevance to this work, 

Chapter 3 presents two views of a diagnostic problem, In the 
first view, diagnosis is considered as a problem in pattern recog- 
nition, The implications and limitations of this view are examined, 
Then the problem of diagnosis is formulated as a sequential decision 
problem, This formulation underscores the computational problems 
associated with the determination of optimal testing strategies, 
Finally, a discussion of heuristic considerations in test selection 
is presented, 

A system for the study of computer-aided diagnosis is des- 
cribed in detail in Chapter 4, This system includes both a diag- 
nostic program and a variety of programs which provide an environ- 
ment within which different diagnostic strategies can be studied, 

The next three chapters are devoted to experiments performed 


with the diagnostic system, Chapter 5 discusses the use of the sys- 


ll 


tem in the diagnosis of primary bone tumors; and Chapter 6, an ap- 
plication of the system to the diagnosis of congenital heart disease, 
A number of other experiments with the system are discussed in 
Chapter 7, Chapter 8 presents a discussion of the results of the 


research and delineates some areas for further investigation. 


Chapter 2 


LITERATURE SURVEY 


A, Diagnostic Programs 


In recent years, there has been an increasing amount of work 
done on various aspects of diagnosis, Some of this work has been 
aimed at the development of computer programs to perform particular 
diagnostic tasks, Other work has been more oriented toward the 
study of human diagnosticians and the strategies they employ, A 
brief survey of this work is presented in this chapter, Examples 
of computer programs for diagnosis are discussed, Of particular 
interest are the diagnostic strategies and models employed by such 
programs, Finally, some broad views of diagnosis and its attendant 
difficulties are considered, 

By far the greatest concentration of research in computer- 
aided diagnosis has been focused in the area of medical diagnosis. 

A number of programs have been written which are capable of perform- 
ing diagnosis in particular medical areas, These programs, as a 

rule employ a Bayesian analysis of attributes based on a disease- 
attribute probability matrix for the given set of diseases considered, 
That is the programs compute the probability of disease D given 

the set of attributes A as follows 

P(D) PCA/D) 


“x P(D) P(A/D) 
D 


P(D/A) 


12 


where P(D) is the a priori probability of D. 

P(A/D) is the conditional probability of A given D, 
The use of a disease-attribute model and Bayesian inference was ad- 
vocated by a number of researchers as early as 1959 (RI, R2, R3, R4,). 
While other means of inferring diseases from their attributes were 
suggested at this time (R5, R6), the Bayesian approach has proved 
the most widely used, In certain areas the use of analog computers 
has been explored, but this work will not be reviewed here, 

In recent years, computer programs incorporating the Bayesian 
model have been developed for problems of heart disease (R7, R8), 
Thyroid disease (R9), epigastric pain (R10), Cushing's syndrome (R11) 
and others, Some of these programs have enjoyed striking success in 
attaining levels of performance comparable to that of the expert hu- 
man diagnosticians, For example, a Bayesian analysis of 268 cases of 
patients with one of three thyroid problems yielded the accepted diag- 
nosis in 96% of the cases, (R9). In a similar analysis of acquired 
valvular heart disease patients, a computer program correctly identi- 
fied 96% of the problems, (R7). In both cases this level of per- 
formance compares favorably with that attained by experienced diag- 
nosticians, 

In order to provide a more detailed view of the use of Bayesian 
analysis in computer-aided diagnosis, two studies will be reviewed 
here, The first is the diagnosis of congenital heart disease; and 
the second, the diagnosis of thyroid function, 


In a series of papers (R12, R13, R14), Warner, Toronto, and 


13 
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Veasy have reported on the development and use of a computer program 
for the diagnosis of congenital heart disease, This program employs 
fifty-seven possible attributes to classify patients into thirty-five 
different disease classes, The basic strategy employed by the pro- 
gram is the use of Bayes' rule to obtain the posterior conditional 
probabilities for the different diseases given a particular set of 
attributes, The necessary a priori disease probabilities and condi- 
tional probabilities of attributes given disease were derived from 
statistical studies of a large humber of known congenital heart di- 
sease patients, In certain instances, the statistical information so 
obtained was deemed inadequate and the probabilities involved were 
estimated from 1) the available literature and 2) consideration of 
the pathologic physiology of the disease, The program takes into 
account the significance of attributes which are absent as well as 
those which are present, Thus, the absence of cyanosis is significant 
in the diagnosis, The program is also designed to account for cer- 
tain mutually exclusive sets of attributes, For instance, if one 
of a set of mutually exclusive attributes is present, it would be in- 
correct to consider the absence of the other attributes in the set 
as additional information in the diagnosis, 

The program is gsed in the following way. For each patient ex- 
amined, the examining physician determines the presence of absence 
of the required attributes, When the examination has been completed, 


the information obtained is punched on cards. and fed to the computer 
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in the field, Furthermore, the accuracy of the computer 

diagnosis is still improving with refinements in the data 

matrix, (R-12) 

Overall and Williams (R-9) developed a computer program for the 
diagnosis of thyroid function, The object was to classify patients 
into one of four classes: 1) no thyroid disease, 2) hypothyroidism, 
3) enthyroidism or 4) hyperthyroidism, By analyzing 879 cases, the 
authors obtained a disease-probability matrix which included 21 in- 
dices of thyroid function, Although over 800 cases were involved in 
the analysis, not all of the 21 measures were available for each 
case, Relative frequencies of each attribute were based on the num- 
ber of cases in which the necessary data were available. Independence 
of attributes was assumed, although the authors note that this assump- 
tion is suspect, 

In an extensive series of tests, the program performed extremely 
well, According to the authors 

+ « » computer diagnoses agreed with the clinical diagnoses 

in over 96% of the cases in which anything like complete 

data were available, (R-9) 

Both of these examples of computer-aided diagnoses lend credence to 
the belief that Bayesian attribute-disease models of diagnosis may 
prove extremely useful in a whole range of medical applications, 

As noted earlier, not all applications of mathematical methods 
to medical diagnosis have been founded on Bayesian inference, An 
interesting example of a different view of the problem involves 
considering a point in an n- dimensional space (where n is the 


number of attributes), From past experience with diseases, one can 


consider each disease as representable by a class of points in the 
space, The diagnosis of the current disease is derived from a 
consideration of the "closeness" of the corresponding point to the 
classes for each of the known diseases respectively.! In a recent 
paper (R-7), Lerner discusses the use of such an approach in the 
recognition of handwritten letters and the detection of oil-bearing 
strata in petroleum geology, In the latter problem (another type 

of diagnostic problem), he reports that a program based on this 
method far surpassed the performance of the most experienced experts, 
He then advocates the application of this method to problems of 
medical diagnosis and asserts that the possibilities of this approach 
"considerably exceed those of doctors-diagnosticians," 

While this method differs markedly from that employed in the 
two medical applications above, it shares with them a very important 
limitation, In Chapter 1 it was suggested that the diagnostician 
performs two major tasks in his problem-solving, The first task is 
the interpretation of attributes manifested by the system being diag- 
nosed, An equally important task is the selection of an appropriate 
testing strategy. All of the programs above map a set of attributes 
into a diagnosis in one stage, There is no test selection function 
performed in any of these programs, As a result, all the data which 
are to be employed by the program must be collected before the pro~ 


gram is invoked, There is no opportunity for selective testing based 
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Chapter 3, . 
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on an analysis of an incomplete set of attributes, Thus, it may 
happen that the cost of determining a number of attributes (for ex- 
ample, by taking an X-ray) is incurred unnecessarily, While this 
may not be a major problem in the particular areas discussed above, 
it is easy to think of situations in which this approach would be 
highly undesirable, Consider, for example, the computer-aided diag- 
nosis of diseases from a group which exhibit clusters of relatively 
disjoint attribute patterns, The approach outlined above required 
the determination of a full set of attributes to be made available 
to the program, Since only a small subset of the set of all attri- 
butes is necessary for a diagnosis, many attributes are unnecessary 
in any particular application, If the cost of obtaining these un- 
necessary attributes is high, then the diagnostic procedure will be 
less than satisfactory, This is because the quality of diagnosis 
should reflect its cost as well as its accuracy, As Lusted has ob- 
served (R-17), 

A great many medical diagnostic tests have been developed to 

supplement the patient information obtained from history 

and physical examination. These tests vary greatly in the 

amount of discomfort to the patient, complexity, and cost, 

It is obvious that diagnostic tests should be kept to a 

minimum, 

It seems that a more satisfactory solution is to permit the 
diagnostic program to operate sequentially, choosing tests for the 
user to run based on a continually updated view of the problem, 


The program could engage in a dialogue with the user as it performs 


both the inference and test selection functions of diagnosis, The 
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testing strategy evolved by the program should reflect the informa- 
tion derived from the attributes observed to date, past experience 
with similar systems, the cost of tests, and the relative seriousness 
of various disease states, Part of the research reported in this 
thesis is aimed at developing a program which satisfies these require- 
ments, 

Less has been done with computer-aided diagnosis in other areas, 
One problem which has received attention, however, is the diagnosis 
of faults in a computer, Although the problems here are not well 
understood at present, recent research (R-18) shows considerable 
promise, Significant results pertaining to the selection of an op- 
timal set of diagnostic tests have been obtained (R-19), but they are 


restricted to the case of a single fault. 


B, Perspectives on Diagnosis 


One of the chief motivations for this research is belief that 
a computer is potentially a very useful tool to be employed in di- 
agnostic problems, The need for such a tool becomes apparent when 
the difficulty of particular diagnostic problems is considered, 

A considerable portion of the effort expended in implementing 
computer programs is devoted to program debugging, As programming 
applications become increasingly sophisticated, the complexity of 
the associated problems of debugging increases at an equally rapid 
rate, The tremendous effort required to debug a large operating 


system is a testament to the magnitude of the diagnostic problem in- 
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involved, This is so even though many of the programmers involved 
in such an effort are experts, 

The non-expert who ventures into the world of programming 
also faces many diagnostic problems, Often the magnitude of these 
problems relative to his limited programming skill and experience 
is such as to prevent him from effectively using the computer in 
his particular research, In both these cases, there is a need for 
an improved diagnostic facility, Research into the potential use- 
fulness of diagnostic computer programs seems especially appropri- 
ate in this context, 

Much the same situation exists in medicine, although here 
there exists more explicit evidence of difficulty of problems in 
medical diagnosis and the need for new aids in the problem-solving 


process, Physicians receive extensive training in their profession, 


_and they devote considerable efforts to the development of their 


diagnostic acumen, For all their training, however, the difficulties 
of the diagnostic problems confronting them have resulted in a sur- 
prisingly low level of performance, In a recent research report of 
the United States Public Health Service entitled "Completeness and 
Reliability of Diagnosis in Therapeutic Practice," the author con- 
cludes from an extensive study 

On the basis of available evidence, I estimate if we re- 

gard all diagnosable diseases at a given time that are con- 

sidered of significance for current health as 1, the num- 

ber of therapeutically determined diseases constitute 


numerically 0.4, Of this 0.4 nearly half are conditions 
diagnosed incorrectly, This suggests that correctly 
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Chapter 3 


TWO VIEWS OF DIAGNOSIS 


This chapter concerns the theoretical framework for the 
study of computer-aided diagnosis, Here the nature of the diag- 
nostic problem is examined and the model for the problem is de- 
veloped, Two views of diagnosis are considered, The first view 
is that of diagnosis as a pattern recognition problem, This con- 
sideration brings into focus those features of the diagnosis which 
distinguish it from the "classical" pattern recognition problem, 
The second view involves analyzing diagnosis as a problem in sequen- 
tial decision-making, The problems arising from this formulation 
are explained and various means of circumventing these problems 
are discussed, The view of diagnosis as sequential decision- 
making is the one taken for this research and so this discussion 
leads directly to the specification of a computer program for per- 
forming general diagnosis, 

In the following chapter, a discussion of a program to perform 
general diagnosis is presented within the framework of the program 
actually implemented as part of this research, Each of the major 
logical functions of the program is discussed in turn with the em- 
phasis on the way in which these functions match the requirements 
of a diagnostic process, In a very real sense, the program can be 


taken as a statement of an overall diagnostic strategy for computer- 


22 


23 


aided diagnosis, 


A. DIAGNOSIS AS A PROBLEM IN PATTERN RECOGNITION 

Consideration of the diagnostic problem as a pattern recog- 
nition focuses attention on some of the more significant aspects 
of the problem, Also it is quite natural to conceive of diagnosis 
as a pattern recognition problem, The observable attributes associ- 
ated with the system of interest in a diagnostic problem do consti- 
tute a pattern which is the direct evidence upon which a classifi- 
cation decision is based, Thus a medical diagnostician confronted 
with an ailing paitent employs his observations of the patient's 
symptoms and signs in conjunction with his experience and training 
to deduce the nature of the patient's problem, While there are 
many features which are shared by the diagnostic problem and a wide 
variety of particular pattern recognition problems, there are addi- 
tional constraints on the former which add to its complexity, The 
purpose here is to explore both the similarities and differences 
between the diagnostic problem and the "classical" pattern recog- 
nition problem, 

The classical pattern recognition problem is fundamentally one 
of recognizing class membership and establishing decision criteria 
for measuring membership in each class, Given a set of pattern 
classes the problem is to assign a new pattern to one of the classes, 
For example in the recognition of handwriting, knowledge of the gen- 


eral properties of individual letters is utilized in the determina- 
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tion of the identity of that segment of handwriting which is cur- 
rently of interest. The individual pattern classes may be known 
in a variety of ways ranging from a set of representative patterns 
to a functional characterization of the probabilistic process by 
which patterns of the class are generated, Im general, a pattern is 
comprised of a set of features; each feature being represented by 
some numerical value, In the handwriting recognition problem, an 
unknown letter could be represented by numerical values for such fea- 
tures as the height, number of loops and the number of intersections 
the letter makes with certain reference lines, Such a representa- 
tion leads quite naturally to the representation of a pattern as an 
n - dimensional vector where n is the number of features which are 
taken to be relevant to the classification problem, 

Hence, each pattern class can be conceived of as a set of 
points in an n - dimensional space, Similarly, any pattern which 
is to be classified can be represented as a point in the space 
(provided, of course, the same set of features obtains), The problem 
of classifying a new pattern sample involves determining the "close- 
ness" of the sample to each of the respective classes, For instance, 
we may decide a certain letter is an "e" because it more closely re- 
sembles representatives of the class of known "e's" than representa- 
tives of other classes of letters, In the n - dimensional space, 
this corresponds to measuring the distance (in some abstract sense) 


between the point denoting the new pattern and those representative of 
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the various classes, The problem of establishing criteria upon which 
the "resemblance" of a particular letter to the class of letters 
known to be "e's" is but one instance of the general problem of de- 
ciding exactly how the "closeness" of a sample to various classes is 
to be established, For a given application, the determination of an 
appropriate metric is a fundamental problem of pattern recognition, 
Consider the schematic of a pattern recognition problem pre- 
sented in Figure 1, Here two pattern classes are of interest, 
classes A and B. Im this case, there are two features in the pat- 
terns and an orthogonal coordinate system corresponding to these fea- 
tures is shown, Notice that in this simple example all members of 
class A are "closer" to all other members of class A than to any 
member of class B and vice versa, Unfortunately, this condition 
does not hold in general, The more common case is to have "close" or 
intersecting pattern classes, Members of a class can be closer to 
members of another class than to certain other members of the same 
class, For example, some handwritten "e's" look very much like "i's" 
and vice versa, A schematic of intersecting pattern classes is pre- 
sented in Figure 2, The problem of recognizing the pattern x in 
these figures involves establishing a metric which can be employed to 
decide whether x is "closer" to the class A or the class B (or in 
some cases deciding that x is a member of neither A nor B), The 
actual decision regarding the identity of x can be based on the cost 


of misclassification as well as the chosen metric, 


nn nr i 


26 


Figure 1 


Two Pattern Classes 


Figure 2 | 


Intersecting Pattern Classes 
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When the pattern classes are dangsenety close or intersecting 
in the space, recognition is more difficult, In some cases matters 
can be improved by devising class separating transformations, Such 
a transformation has the property that the classes resulting from 
the application of the transformation to the original classes are 
more separated from one another in the transform space, Figure 3 
represents the effect of a class-separating transformation on classes 
A and B, The particular transformation will depend on both the 
characteristics of the classes to be transformed and the constraints 
placed upon the transformation, Suffice it to say here that trans- 
formations of this type can be derived by solving constrained op- 
timization problems, Given such a transformation, the pattern to 
be recognized is first transformed and then its "distance" from 
each of the transformed classes is measured, It is this distance 
in the transform space which is incorporated in the classification 
decision rule, 

The problem of diagnosis has much in common with the pattern 
recognition problem discussed above, The pattern classes in the 
pattern recognition problem correspond to the system states in the 
diagnostic problem, and there is a similar analogy between particular 
patterns and sets of attributes, The object of diagnosis is to class- 
ify a set of attributes as being a manifestation of a particular sys- 
tem state, Again, the notions of an n - dimensional space and vector 
representations of attribute sets is suggested, There is an important 


difference between diagnosis and the pattern recognition method out- 


Figure 3 


Transformation of Pattern Classes 
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lined above, In the latter, it was assumed that a pattern to be 
recognized is given as a point in the sample space, This implies 

a complete specification of the corresponding vector. In the usual 
diagnostic problem, the pattern of attributes is incompletely speci- 
fied, There exist means for obtaining the values of unspecfied com- 
ponents of this vector (tests which can be run, etc,), but in general 
there is a cost associated with the use of these means, These costs 
make it advantageous to analyze the diagnostic problem sequentially 
and to make decisions based on an incompletely specified attribute 
vector, Doctors, for example, make diagnostic decisions without 
performing all possible tests on the patient, 

Thus, in the diagnostic problem, one is concerned throughout 
with subspaces of the sample space, The dimensionality of the sub- 
space which contains the pattern vector is reduced by obtaining pre- 
viously unspecified values for certain pattern features, In general, 
each value so obtained reduces the dimensionality of the subspace in 
which the point corresponding to the fully specified attribute set 
must lie, Because of the costs associated with the tests for particu- 
lar attributes, a good diagnostic scheme must include some means for 


assessing the expected value of a test in determining the class to 


lnote that this distinction between pattern recognition techniques 
and diagnostic techniques is not a necessary one, Certain pattern 
recognition schemes have employed sequential methods while most medi- 
cal diagnosis programs have avoided sequential analysis entirely, The 
distinction, however, does have appreciable generality. 


which the attribute vector belongs, While the sequential nature of 
the diagnostic process complicates its realization, it also offers 

a potential advantage of the pattern recognition scheme described 
above, Although an attribute vector may be incompletely specified, 
the subspace corresponding to it may include only one class, In 
such a case it may be possible to make the classification decision 
at that point without investigating the remaining attributes, This 
reduction in the amount of the processing required for a classifica- 
tion decision is especially significant when many of the system 
states are represented by disjoint subspaces in the n - dimensional 
sample space, This reduction can be obtained only if the diagnostic 
scheme incorporates some stopping rule for the attribute sampling 
(or testing) process, 

So while the pattern recognition problem and the diagnostic 
problem have a number of features in common, there are significant 
differences between the strategies indicated for their solution, The 
former problem concerns the classification of a fully-specified 
vector into one of a number of known classes, The latter problem 
is equally one of classification, but the initial specification of 
the vector is generally incomplete. Part of the problem is to as- 
certain which tests to run (at some cost) to obtain a more complete 
specification of the vector, Decisions based on an incompletely 
specified vector are the rule rather than the exception, Note, 


however, that there may well be inherently close or intersecting 
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classes in the diagnostic problem as in the pattern recognition 
problem, 

One aspect of the pattern recognition problem which was not 
discussed above was that of choosing the coordinate system for the 
sample space, This has a direct and significant analogy in diagnosis, 
In the discussion of pattern recognition, it was assumed that the 
pattern features were given. The efficiency and the accuracy of 
the recognition scheme often can be improved by the selection of a 
new coordinate system (set of features). The problem of establish- 
ing the coordinate system is often termed the pattern detection 
problen, 

Thus, for example, in Figure 1 the dotted coordinates are in 
a sense more efficient, for they permit the characterization of 
classes A and B solely in terms of one coordinate, Again general 
mathematical techniques are known for establishing "good" coordi- 
nate systems for a number of problem types, 

Clearly, a similar situation obtains in diagnostic problems, 
Generally speaking, the attributes considered in diagnostic problems 
are chosen without any particular regard for the efficient separation 
of pattern classes, It is apparent, however, that there is potential 
value in conducting such an analysis for a given problem area, In 
certain areas, especially in a medical diagnosis, there has been an 
increasing awareness of the importance of the proper choice of pattern 


features; a number of articles on the "taxonomy of disease" 
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have appeared in the literature, While this problem is an extremely 
interesting one, it is beyond the scope of this thesis, Here the 
pattern features of attributes for any particular area are taken as 
given, 

This discussion provided only a brief overview of pattern recog- 
nition and its relation to diagnosis, The particular type of pattern 
recognition which constitutes diagnosis will be explored in con- 


siderable detail in other sections of this work, 


B, DIAGNOSIS AS A SEQUENTIAL DECISION PROBLEM 

In this section, the problem of diagnosis is formulated in 
terms of statistical decision theory. This formulation is in very 
general terms, but it suggests a number of the factors which compli- 
cate particular diagnoses, ‘In many areas of diagnosis, attention is 
focused on a system, In medicine the system is a human being; in 
program debugging, a computer program, The object of the diagnostic 
problem is to determine the state of the system (e.g, the disease in 
the person or the error in the computer program), This state is one 
of a finite but perhaps quite large number of possible states, In- 
formation about the state of the system can be obtained by performing 


a variety of tests on the system , Information obtained from testing 


lin recent years, there has been much medical work directed 
at developing specific tests for diseases, Thus a particular at- 
tribute (test result) may indicate exactly one disease, 
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coupled with experience with other diagnostic problems is employed 
by the diagnostician in his attempt to deduce the state of the sys- 
tem, In this work, the goal of diagnosis is taken to be the deter- 
mination of the state of the system of interest, It is assumed that 
knowledge of the system state will greatly facilitate further (non- 
diagnostic) action, For example, the identification of the state 
of a patient as "tuberculosis" may lead directly to a course of 
treatment, The system under consideration here is a finite state 
machine, The diagnostician knows about all the states of the machine 
in the sense that he has available probability distributions which 
characterize the response of the machine to certain tests given the 
machine state, In particular, this information relates attributes, 
the results of the tests, to particular system states,! At the 
outset of the problem, the machine is in a particular, but unknown 
state and the task of the diagnostician is to employ the available 
tests to obtain information about the identity of that state, Tests 
are assumed to be free from error and it is further assumed that they 
do not alter the state of the system, 

Associated with each test is a cost of applying it to the system 
(called the testing loss) and thus it is advantageous to make a de- 
cision about the state of the system based on a limited number of 


tests, On the other hand there is a decision loss associated with an 


lan attribute is binary-valued, That is, each attribute is 
either present or absent, A test is used to determine the presence 
or absence of some number (perhaps greater than one) of attributes, 
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incorrect decision, The loss resulting from each particular decision 
about the unknown state as a function of the actual state is given 
by a loss function for the problem, For example, the loss resulting 
from the decision that a tumor is benign when it is in fact malignant 
is very costly and a diagnostic procedure for tumors should take 
cognizance of this fact, In general, the possibility of loss for 
an incorrect decision indicates the value of extensive testing prior 
to any decision, The problem is to balance the testing loss and the 
decision loss in a sequential decision function for the problem, 
This function would specify a diagnostic procedure such that the 
total expected loss of the final decision is minimized, The follow- 
ing is a formal statement of this problem, 
1, The states of the Machine M are M;, j=l,n, 
and the current state is denoted by M,. It is assumed 
that M, does not change during the course of the 
diagnosis, 
2 TT = Wi, -- -T is a vector of a priori probabili- 
ties for M,. That is Th = P(M, = Mie) 
and & denotes experience, 
3, Te dtu, --- tr} is the set of available tests, 
4. (ty)q is a vector of length q with each t;eT, It repre- 
-th 


sents a series of tests with test t; being run at the i-—— 


stage, 
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12, 


{s,, -- Spj is the finite set of possible attributes 
for M and the set T, 


(3) 


q is a vector of length q with each S;€éS. It denotes a 
sequential set of attributes, 

d, is a terminal decision and d,€D, where D, is the finite 
set of all possible terminal decisions, 

C((tidg, (Si)q) is the testing loss for a sequence of tests 
(tidg resulting in the attribute sequence (Sidg followed by 
terminal decision at stage qtl. 

PC (Si) g/M;) is the conditional mass function for (Sidg 
given M;. 

PC(tidgsde/ (Si) q) = conditional mass function for the testing 
sequence (tidg followed by terminal decision d, given the 
attribute sequence (Sidg 

LOT ,4,) is the decision loss function. 

O(4/(t5)4,(83)Q) is the sequential decision function to be 


determined, 


Let L, 7,9) = the average decision loss 


LUT 59) = the average testing loss, 


then the problem is to determine @ such that 


L, (7 ,8) + L,(T ,e) 


is a minimum, 


pas ial 
ars 2 ee ZZ LCT 4) OCIS) q (5) g) » PC(Si)g/M5) 
Sie ae 5q t 


12( 1,8) => > PM ie PC(tagsdg/ (Sig) » CCSi) gs (ta gQPC(Si) g/Mj) 
- q 


where Ty is the set of all (tig 


and Sg is the set of all (Sig 


The great difficulty with this problem is not conceptual but com- 
putational, For finite sets of attributes and decisions, the optimal 
solution can be obtained in principle by laying out a decision tree, 
Such a tree includes by two types of nodes--decision nodes and "nature's 
nodes."' Nodes of the former type are characterized by 1) a current 
view of the diagnostic problem as embodied in the probability dis- 
tribution over the states of the system, (This distribution accounts 
for both the attributes observed to date and the a priori likelihood 
of system states in a manner to be made explicit later in this thesis.), 
and 2) a branch emanating from the node for each alternative available 
to the decision-maker at the node, In the context of diagnosis, then, 
there is at each decision node one branch for each possible test which 
can be run and one branch corresponding to a terminal decision, Once 
an alternative branch away from a decision node has been chosen by 
the decision a particular one of nature's nodes is encountered, 

Such a node represents the possible outcomes of the decision cor- 
responding to the branch which leads to the node, Each of these "out- 
come branches" leads to a new decision node, A portion of such a 


decision tree is shown in Figure 4. The node A is a decision node 
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Figure 4& 


Section of a Deci 


sion Tree 
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which is characterized by the prior probability distribution and his- 
tory embodied in the path to the node, There is a branch from this 
node for every relevant test (given the history and TI) as well as a 
branch corresponding to a terminal decision. If a particular test is 
chosen, say test T; in the diagram, a new node (here node B) is ob- 
tained, This node is one of the "nature's nodes" mentioned above, 
There is a branch from this node for each possible test outcome given 
T, and given the state of the diagnosis at B, the conditional proba- 
bility for each attribute branch can be computed. 

If it is assumed that the total number of potentially useful 
test sequences is finite then the entire tree for the diagnosis can 
be specified. By folding back this tree in terms of expected loss, 
one can obtain an optimal decision for every decision node on the 
tree, This problem is amenable to techniques such as dynamic pro~ 
gramming. There is little conceptual difficulty in solving the 
problem, 

The difficulty is the exponential growth of the number of de- 
cision nodes with the number of signs and tests, Since diagnostic 
problems involving large numbers of possible attributes are common, 
it is expected that the problems of searching large decision trees 
contribute a large part of the complexity of specific diagnostic 
problems, One of the major concerns of this research is with the 
development of effective heuristics for this tree searching problem, 


While such heuristics produce sub-optimal solutions, it is possible 
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that the reduction in the size of the search space may more than 
offset this disadvantage. 

As an indication of the potential size of such a problem, con- 
sider the diagnosis of a ten-state, twenty-attribute system. Such 
a case might arise when one was attempting to employ twenty attrib- 
utes to classify a person into one of ten disease groups. Assuming 
that there is a test for the presence or absence of each attribute 
and that each test is run but once, the number of decision nodes in 
the decision tree for the problem can be expressed as 


kay 
nN_ = 2 


(n-k)- 


Where ,N, = the number of decision nodes 


> 
u 


the depth of the tree 


the number of tests, 


i=) 
tt 


For this example, n is 10, and the number of decision nodes ina 
tree of depth k is given by 


2kj0! 


N. = 
OE 0c 


Table 1 gives values of ;oN, for selected values of k. Notice the 
extremely rapid increase of, oN, with k, Also, at any given decision 
node at depth k it is necessary to compare (n-k+l1) decisions (one for 
each of the n-k remaining tests and one for the possible terminal 


decision), Although in many cases such an attribute set is highly 
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redundant, it is often possible that a depth of 5 may be required 
for an optimal decision. In such a case there are still almost a 
million decision nodes, Even in the simple case of a specific test 
for each state, there are n! different decision nodes, where n is 
the number of states, Again the growth of the decision tree with 


n is enormous. 


Table 1 


Growth of Search with Depth 


k =0 105 = 1 
1 20 
3 5,760 
5 967,680 


While there are certain factors in particular diagnostic 
areas which allow the decision tree to be considerably reduced in 
size, the determination of an optimal testing strategy reamins com- 
putationally infeasible for the most part. The value of good 


heuristics is apparent from considerations such as the above. 
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C. HEURISTIC CONSIDERATIONS IN TEST SELECTION 

As previously noted, the problem of obtaining an optimal test- 
ing strategy for a particular diagnostic area generally will be 
computationally infeasible, Many diagnostic areas are character- 
ized by overlapping attribute patterns for different states and highly 
redundant attribute patterns, however, and there is strong motivation 
for developing "'good'' diagnostic strategies, Unnecessary or re- 
dundant tests may exact a high cost which could be avoided by a more 
efficient testing strategy, In certain areas of medicine, tests 
are quite costly and may cause the patient considerable discomfort, 
If such tests contribute little additional information to the 
diagnosis, it is especially important that these tests not be em- 
ployed, A second difficulty is that a poor sequence of tests may 
generate results which, being umnecessary for a diagnosis, simply 
tend to obscure the truly relevant attributes, One approach to 
this problem was mentioned earlier, This approach consists essen- 
tially of sharpening the taxonomy of the problem states, While 
success here can substantially reduce the redundancy in attribute 
patterng, it will not necessarily make the determination of an op- 
timal testing strategy computationally feasible, While the possi- 
bilities of this approach are extremely interesting, they will not 
be considered here, For the purposes of this work, it is assumed 
that in any diagnostic area, the attributes for states are given, 
No attempt is made to improve on the efficiency of the given attrib- 


utes with regard to the characterization of the states, 
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A second approach to the problem of test selection is to de- 
velop heuristics for the selection process, Such heuristics would 
employ only limited segments of the decision tree in evaluating 
the potential efficacy of relevant tests, The general nature of 
the diagnostic problem is such as to offer two distinct means of 
controlling the growth of the number of decision nodes considered, 
The size of the decision tree (the number of decision nodes) de- 
pends on the number of tests considered at any decision node, and 
the depth of the analysis of that tree, By restricting either of 
these quantities, the diagnostician can limit the growth of the 
tree, In this discussion, heuristics which limit the number of 
branches from a decision node will be called breadth-limiting; 
and those which limit look-ahead, depth-limiting, In what follows, 
the set of relevant tests for a particular decision node will be 
taken to mean all those tests which can result in a sign which is 
manifested by at least one state with a non-zero probability in 
the prior for the node, The set of relevant tests is a subset of 
the set of all tests, 

Breadth-limiting heuristics are easily formulated, Perhaps 
the simplest is to limit the number of branches from a decision 
node to some fixed number, If this number is less than the number 
of possible test branches for a given node, then a decision rule 
for selecting (or rejecting) branches must be established, In terms 


of the diagnostic problem, this means selecting a subset of the 
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relevant tests for consideration given a prior distribution for 
the unknown state, 

Heuristics which limit the number of branches from a decision 
node to a certain fixed number have several shortcomings, Principal 
among these is the problem of the selection decision rule. If 
certain tests are to be selected over other tests, then some measure 
of test effectiveness should be employed, That is, one test is 
chosen over another because by some standards the former is more 
promising. The difficulty with this is that almost any reasonable 
measure of expected test effectiveness requires information obtained 
from a look-ahead in the decision tree, To assess the potential 
value of a particular test, one needs to consider the likelihood 
of various test results and the value of these results in improving 
the current view of the diagnostic problem, If this look-ahead is 
performed, the purpose of the heuristic is defeated, A breadth- 
limiting heuristic is intended to select a subset of relevant tests 
without employing a look-ahead procedure, Then this subset is 
subjected to further analysis, 

Since a breadth-limiting heuristic probably should not employ 
a look-ahead to abtain information, the only information upon which 
it should make its decisions is that contained in the current prior 


distribution and the test cost data,! Thus one possible breadth= 


lonis may be overly restrictive, since one can imagine breadth- 
limiting heuristics which employ a priori probabilities, Such heuris- 
tics are not in general very sophisticated, and are not considered here 
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limiting heuristic is "At any decision node consider at most 5 tests 
in order of increasing cost." This heuristic obviously ignores all 
the information embodied in the current prior distribution, and so 
while it limits the breadth of the decision tree, it does not appear 
to be a particularly good heuristic, 

An alternative breadth-limiting heuristic employs the current 
prior distribution to generate the subset of relevant tests which are 
to be considered, For each state there are a number of relevant 
tests, These tests may produce an attribute which is significant in 
the diagnosis of the state, Consider, for example, a problem in 
medical diagnosis in which one of the diseases which currently is 
being considered as the explanation of the patient's problem is 
tuberculosis, Since a chest X-ray is a useful test in the diagnosis 
of this disease, it would be considered a relevant test, On the 
other hand, the absence of any attributes associated with an in- 
jured ankle would exclude an ¥-ray of the ankle from the set of rele- 
vant tests at this stage in the diagnosis, The union of the sets of 
tests relevant to currently possible states is the set of all rele- 
vant tests, By limiting the number of states considered, one can 
limit the number of branches at the decision node, A heuristic of 
this type is "Create the total set of relevant tests from the sets 
of relevant tests for the three most probable states (based on the 
current prior)." In the above example, if tuberculosis were cur- 


rently the most probable disease, the diagnostician might choose to 
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consider only those tests which are relevant to tuberculosis and 
ignore all others, Note that such a heuristic is only potentially 
breadth-limiting, There is no guarantee that any test branches are 
excluded in this way since the same set of tests may be relevant to 
all states currently being considered, Also the actual number of 
branches from a given decision node is not specified and generally 
will vary from node to node, 

Such an heuristic has intuitive appeal, however, because it 
prunes branches corresponding to tests for attributes specific to 
improbable states, If an attribute for an improbable state is also 
manifested by a state which is currently quite likely, however, then 
the appropriate test will be included in the set of those considered, 
The weakness of this heuristic lies in its sensitivity to the current 
probability distribution on the states of the system, This distribu- 
tion can undergo radical change upon the observation of one new 
attribute, Thus, states which were previously unlikely can become 
very probable as a result of one new observation, This phenomenon 
cannot be accounted for by breadth-limiting heuristics based on the 
current prior distribution, In fact, no breadth-limiting heuristic 
which does not employ look-ahead can completely account for this possi- 
bility, A breadth-limiting heuristic of this type is applied at each 
decision level, however, and in some sense it can "recover" from a 
drastic change in the probability distribution, This capability is 
derived from the consideration of the probability distribution at the 


current decision node, Thus, when a state which was formerly improbable 
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at one decision node becomes probable, it will automatically be in- 
corporated in the test selection scheme at the next level. Unfor- 
tunately, this state may not become very likely until a large number 
of tests have been run, If it is the actual state, its probability 
can remain low simply because the "wrong" tests are being run, Thus 
a doctor may fail to obtain a chest X-ray of a patient because it seems 
unlikely that the patient has tuberculosis, when this disease would 
become very probable if only the X-ray were taken, This, of course, 
is a general problem encountered with all test selection heuristics, 
The evaluation of the heuristic involves a comparison of the 
benefits of its tree-pruning power with the losses incurred from the 
sub-optimal testing strategies it produces, In general, a heuristic 
based on the current probability of various system states appears to 
be the most promising form of a breadth-limiting heuristic, but its 
actual value can be determined only in the context of a particular 
diagnostic problem area, For example, in one area a breadth-limiting 
heuristic which restricts the search to tests relevant to the n most 
probable states may prove useful, In another area, tests relevant 
to all states with current probability greater than some threshold 
may be considered, Finally, in certain areas breadth-limiting heur- 
istics may be of no value regardless of the particular specification, 
One of the areas explored in this research is that of evaluating sev- 
eral breadth-limiting heuristics in particular diagnostic problem 
areas, In such an evaluation, the capability of closed diagnosis may 


be particularly valuable, 
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As noted in the beginning of this section, there are two gen- 
eral types of heuristics which reduce the number of decision nodes 
considered in test selection: breadth-limiting and depth-limiting, 
As the name of the latter implies, such heuristics limit the extent 
of the look-ahead in the decision process for test selection, As 
with breath-limiting heuristics, there are several variations of 
the depth-limiting heuristic to be considered, 

Perhaps the most obvious form of the depth-limiting heuristic 
is one which sets a fixed depth of search for all branches of the 
tree, Thus given a particular decision node, the search would pro- 
ceed down all branches from that node to a depth k, where k is a 
fixed number, The information derived from this search would then 
be employed in a decision rule to determine the test to be run next, 
The parameter k is a relative depth, that is at a decision node at 
level p, the search is conducted to a depth of ptk before making 
the decision for level p. An alternative depth-limiting heuristic 
might employ a variable depth look-ahead, Such a heuristic might 
attempt to explore more "promising" branches to a greater depth than 
less promising ones, The difficulty here is to decide which branches 
are promising, It is, in fact, the general problem of heuristic 
test selection all over again, 

There are several problems to be resolved in the development of 
any depth-limiting heuristic, First consider the effect on the de- 


cision process of limiting the depth of search, If the depth is 
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limited to k, then the "terminal" nodes will be characterized by 
probability distributions for the unknown state, (See Figure 4.) 
Since, in general, there will be a number of states with non-zero 
probability at any given terminal node, there must be some way of 
assessing the value of being at the node, One of the major problems 
in the development of depth-limiting heuristics then is the defini- 
tion of measures of the desirability of nodes which do not represent 
a certain diagnosis, 

One way of establishing the value of a node is suggested by 
the presence of a loss function, The value of the node can be ob- 
tained by assuming a decision about the unknown state is to be made 
there, Then the prior distribution for the node and the loss func- 
tion can be employed to find the expected decision loss for the 
node, From this loss the value of the node is derived, While 
this measure seems to be a natural one, it is not without its weak- 
ness, The problem with the measure is that it is based on an 
assumption which is generally untrue, In most cases, one will not 
make terminal decisions at the nodes which are "terminal" for one 
state in the look-ahead, For example, if the search depth is 
limited to 2, the value measure assumes that a terminal decision 


will be made two tests from this point, Since the actual terminal 


tan additional assumption should be noted here, This is the 
assumption that given the prior distribution, the minimum expected 
loss decision is made, 
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decision may not be made until many tests have been run, this 
measure distorts the value of tests considered for the current level, 
The problem is that the values of the loss function at the decision 
nodes of a given level may bear little relation to the values of 

the best testing strategies which include these nodes, The poten- 
tial effectiveness of this "loss function" measure is difficult to 
assess, The expectation is that it depends upon the particular 
problem area in which the measure is employed, 

A second problem with this heuristic is its potential sensi- 
tivity to the actual loss function employed, If the heuristic is 
very sensitive to the loss function then uncertainties as to the 
true nature of this function may result in testing strategies which 
are decidedly sub-optimal, The problems of accurately assessing 
the loss function for a particular application will be discussed 
later in this thesis, 

The above discussion of breadth-limiting and depth-limiting 
heuristics purposely considered the two independently in order to 
make clear the considerations involved, The motivation for such 
heuristics in test selection is the desirability of reducing the 
number of decision nodes considered, Since the number of decision 
nodes is dependent on both the breadth and depth of the search, 
the heuristics employed in an actual problem will interact, Gen- 
erally speaking, the depth of the search can be increased only at 
the expense of the breadth, because there is a constraint on the 


total number of nodes to be considered, The particular balance of 
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these two heuristics may significantly affect the effectiveness of 
the test selection process, An additional complication is introduced 
by the possibility of changing this balance during the course of the 
diagnosis when many states are possible, It may be desirable to 
limit the depth and allow full breadth, This is particularly true 
if the prior distribution is quite diffuse, As the diagnosis pro- 
gresses and certain states are eliminated from further consideration 
the breadth of the tree may be reduced and the depth of search may 
be increased correspondingly, The relation between the depth and 
the breadth of the search is an important matter for investigation 
in the development of heuristic test selection schemes, 

More of the practical considerations involved in developing 
heuristics will be discussed in a later section describing the 
heuristics employed by the diagnostic program and their relative 


effectiveness, 


Chapter 4 


A DIAGNOSTIC SYSTEM 


The considerations outlined in the previous chapter led to 
the design and implementation of a diagnostic system, This system 
is composed of three major parts, The first is a set of programs 
which perform the actual diagnostic function, The second is a set 
of programs which facilitate the study of a variety of diagnostic 
problems and strategies, The third part of the system is the informa- 
tion structure which contains all the relevant information which 
these programs employ in performing diagnosis for a given problem 
area, While the content and, to some extent, the nature of the 
information structure vary with the particular application, it is 
convenient to consider this structure as a third general part of the 
diagnostic system, These three aspects of the diagnostic system will 
be discussed in detail in this chapter, 

The diagnostic system is currently operating on the Project 
MAC time-sharing system at the Massachusetts Institute of Technology, 
The diagnostic system is designed to exploit the inter-active capabili- 
ties of the time-sharing system, The programs of the diagnostic 
system are written in MAD and FAP, They make very extensive use of 
the SLIP-MAD system developed by Professor Joseph Weizenbaum of M,I,T, 
The SLIP-MAD system (hereafter referred to as SLIP) is a set of list 


processing functions embedded in the host language MAD, Because 
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discussions of SLIP are available elsewhere (R=20), only a brief out- 
line of the system is given here, 

The basic data structure employed in the SLIP system is a SLIP 
List. A SLIP list is a list composed of cells where a cell is a pair 
of adjacent words of storage, The first word of the pair is divided 
into an identifier field, a link-left field and a link-right field, 
Each cell in a SLIP list contains a forward (right) link and a backward 
(left) link, SLIP lists are symmetric in the sense that lists have 
no particular orientation, the top and bottom of a list are equally 
accessible, The identifier is used to indicate the type of element 
stored in the second word of the cell, This element is referred to 
as the datum, An example of a simple SLIP list is given in Figure 5, 
Notice that any cell may contain an actual datum rather than a symbolic 
designation for the datum, 

Every SLIP list contains a special cell known as the header 
of the list, This cell contains the address of the first cell on 
the list in its right-link field and the address of the last cell on 
the list in its left-link field, Any storage location which contains 
the address of a list header in both its address and decrement fields 
is said to contain the name of that list. A SLIP list structure can 
be defined as a SLIP list whose data terms may themselves be names 
of SLIP lists, 

There may be associated with any SLIP list a description list 


or DLIST, If a SLIP list possesses a DLIST, the address of the header 
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A Simple Slip List 
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of the DLIST is contained in the left-link of the datum of the 
header cell, The DLIST, which is itself a SLIP list, is used to 
store data pairs, A variety of SLIP functions are available for 
creating and accessing these pairs, 

The SLIP library is a set of functions for manipulating SLIP 
lists, Typical functions permit the reading or searching of lists, 
additions to or deletions from lists, and the creation or erasure 
of lists, SLIP maintains an available space list, and the system 
includes an automatic garbage collection facility, 

Because the SLIP library consists of compiled subroutines 
which can be invoked from MAD or FAP programs, SLIP programs run 
at object speed, The fact that SLIP is embedded in an algebraic 
language, MAD, means the full arithmetic and logical capability 
of the latter is available to the programmer in a list-processing 
application, These two features make SLIP a convenient language to 
use in the implementation and debugging of a large list-processing 
application such as the diagnostic system developed in this research, 
For this particular application, the need for both the flexibility 
of list-processing and the algebraic power of MAD is well served by 


the SLIP-MAD system, 


A, THE INFORMATION STRUCTURE FOR THE DIAGNOSTIC SYSTEM 
The manner in which the information relevant to a particular 
diagnostic problem area is organized has a considerable effect on 


the capabilities of the diagnostic program, The information contained 
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in this structure for a particular application constitutes the "ex- 
perience" which the diagnostic program brings to bear on problems, 
This experience includes relationships between observable attributes and 
states of the system to be diagnosed, For example, in an area of 
medical diagnosis, the information structure would contain the re- 
lationships between signs and symptoms and the appropriate diseases, 
Also included in the structure is information about the tests which 
are relevant to the given diagnostic area and their associated costs, 
Because of the probabilistic nature of many .of the attribute-state 
relationships as well as other important relationships, the informa- 
tion structure must maintain a large number of individual probabilities, 
The general content of the information structure will be explained 
below, 

The large number of state, attributes, and tests encountered 
in many diagnostic areas places a premium on efficient searching of 
the information base during a diagnosis. The efficiency of search 
can be maintained at an acceptable level only through the proper organi- 
zation of the relevant information, 

A number of questions were considered in the design of the 
information structure currently employed by the diagnostic system, 
One of the principal questions was that of what information should 
be maintained in the structure, To a large extent, the particular 
diagnostic problem under investigation here determined the answer to 
this question, Since the model of diagnosis makes reference only to 


states, attributes, tests and various probabilities, these factors 
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constitute the basic information blocks in the structure, Another 
question is how, given che basic information blocks, these blocks 
should be related in order to facilitate access by the diagnostic 
program to the relationships which are significant in the deductive 
process of diagnosis, For example, the following questions typify 
the types of demands made on the structure, 

* What are the symptoms of pneumonia? 

* Which diseases exhibit a rash on the arms as an 
attribute? 

* What is the probability that a patient will have a 
temperature greater than 103° given that he has 
pneumonia? 

The information structure described here was developed through 
the consideration of a number of alternative forms, although there 
obviously are other forms which might serve as well, To a certain 
extent, the information structure reflects the use of the SLIP 
system by the diagnostic program, For example, the information 
structure is a SLIP list structure, While in certain instances 
this results in inefficient utilization of main storage, this dis- 
advantage was more than offset by the convenience of being able to 
employ the full SLIP library in the development of the diagnostic 
system, 

A basic information block in the structure is either a state, 
an attribute, or a test. Each of these basic blocks is represented 


by a SLIP list in the information structure, In what follows these 
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blocks will be referred to as state lists, attribute lists, or test 
lists, A typical state list is depicted in Figure 6; in this instance, 
the state list corresponding to pneumonia in a medical diagnosis 
problem, The list name of each attribute list relevant to pneumonia 
appears on the state list for this disease, There are two data pairs 
on the DLIST of each state list, The stored attributes are the a 
priori probability of the state and the print name of the state, The 
latter is the name by which the user of the program makes reference 
to the state, In order to facilitate the retrieval of the state list 
corresponding to a particular print name (as, for example, when the 
user makes a request for information about the disease pneumonia), 
all the state lists are grouped on a number of hash lists, Each hash 
list is a sublist of a list calted the master state list, The re- 
trieval of the state list corresponding to a particular print name is 
effected as follows: First a SLIP function is used to map the given 
print name onto the integers 0 to N-1, where N is the number of hash 
lists on the master state list. If the integer K-l results from this 
mapping, the Kt) hash list is searched for a state list with the de- 
sired print name, Since the same hashing function is employed in the 
creation of the master state list, the appropriate list will be found 
if one exists, Roughly speaking, this technique reduces the average 
search time for such requests by a factor of 1/N as compared to a 
search in the absence of hash lists. 

An attribute list includes the list names of all the test lists 


corresponding to tests which can result in the given attribute, 
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The DLIST for an attribute list contains a data pair for the attrib- 
ute print name in addition to a special data pair for a member list, 
The member list for an attribute list is a standard SLIP list which 
contains the list name of each state list on which the name of the 
attribute list appears and the corresponding probability of the at- 
tribute given the state, Continuing the example above, Figure 7 de- 
picts the attribute list for the attribute "fever,"' As in the case 
of the state lists, each attribute list is a sublist of a hash list, 
and each of these hash lists, in turn, is a sublist of the master 
attribute list, 

A test list contains the cost of the test and a DLIST, The 
DLIST contains the print name for the test and a member list for the 
attribute lists which include this test. In Figure 8 a simple test 
list is shown with a single cost (independent of state) and a deter- 
ministic member list, This is the form of test list used in this 
research although it would be relatively easy to make it more com- 
plex, As above, each test list is a sublist of a hash list, which 
is in turn a sublist of the master test list, A schematic of a por- 
tion of the information structure is shown in Figure 9, 

The presence of two-way links between attributes and states and 
attributes and tests results in a highly associative information 
structure, This associative property facilitates the accessing of 
information pertinent to a diagnosis, Thus a search for attributes 
given state and a search for states given attribute are equally effici- 


ent. Similarly the accessing of possible attributes resulting from a 
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particular test is made straightforward by the presence of the member 
list, 

One example of the importance of this associative aspect of the 
information structure is its use by the diagnostic program in the 
initial "pruning" of the space of possible diagnoses in response to 
the observation of initial attributes, Generally, these initial 
attributes are presented as the user's statement of the problem, 

For the program to operate in a reasonably efficient manner, it must 
use this initial statement of the problem to develop a drastically 
reduced set of states for further consideration, This is directly 
analogous to the "pruning" employed by a doctor when upon the observa- 
tion of a few initial signs or symptoms, he reduces the list of di- 
seases he considers as possible causes of the problem to a very small 
number relative to the set of all diseases, The diagnostic program 
would employ the member list for a given attribute list to rapidly 
determine the set of all diseases which were known to exhibit the 
corresponding attribute. While this reduction of the search space 

is crucial to the success of the program, it must not be irreversible 
if the program is not to be led astray by spurious information or 
noise, Since it is unreasonable to expect that those who prepare the 
information structure can anticipate all variations in attribute pat- 
terns for a given state, it is expected that the program at times will 
be confronted with problems involving attributes which are not rele- 


vant to the principal problem, The strategies employed by the pro- 


65 


gram and the nature of the information structure have a strong effect 
on the program capability in such a problem environment, 

The information structure currently employed by the diagnostic 
program associates with each state only those attributes which are 
relevant in the diagnosis of that state, Thus there would be no 
association between the state "tuberculosis" and the attribute "sore 
thumb" in the information structure for medicioe,* The advantage of 
this is that the size of the information structure is limited, Thus 
while there may be many attributes, only a subset is associated with 
any state, As will be discussed later, this creates problems in 
performing diagnosis in a noisy environment, Certain routines asso- 
ciated with the diagnostic program are responsible for making de- 
cisions about the significance of the attributes observed in a diag- 
nosis, The function of these routines is also the subject of a 
later section, 

The discussion of the information structure to this point has 
implied that the attributes for a given state are taken to be in- 
dependent. Since in many cases the assumption of attribute independence 
is not justified, it is necessary that inter-attribute dependencies 
be representable in the structure, This capability is available in 


The current program through the use of clustering routine, the 


tsince the program does not determine what information is in- 
cluded in the structure, the user can associate any attributes and 
states, The point is that certain associations are not expected, 
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relation-definition routine, and the relation interpreter, 

In order to provide a general capability for dealing with 
inter-attribute dependencies, the diagnostic program must be able to 
cope with a variety of relationships among attributes, The import- 
ant relationships most likely vary from one diagnostic problem 
area to another. It does not seem advisable to attempt to catalog 
these relationships within the program itself, since it is extremely 
difficult to predict just which relationships will be required, Also, 
if the relationships are incorporated within the program itself, it 
is difficult to introduce now ones as they become of interest in a 
particular problem area, 

What is required then is a flexible facility for the program 
to accept new relationships and having so accepted a relationship, 
to incorporate it correctly in the inference process of diagnosis, 
In an attempt to provide this facility, the diagnostic program pro- 
vides the user with the means to define a variety of relationships 
among attributes. A relationship is defined by specifying as a 
Boolean function the conditions under which the relationship is 
true, This function is employed by the diagnostic program whenever 
it is necessary to determine whether the relationship is satisfied 
for a particular state, 

Consider, for example, the case in which it is necessary to 
account for the time of the appearance of certain attributes of a 


particular disease, Imagine that for the disease in question the 
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attribute "rash" appears two days after the appearance of "fever," 


Let the function BEFORE accept five arguments and be defined as 


BEFORE (A1,A2,A3,A4,A5) = 


(EQ (MINUS (CHAR Al A2) (CHAR A3 A4)) AS) 


Here EQ, MINUS, and CHAR are system primitives (defined by the 
diagnostic program), The function CHAR is used to retrieve charac- 


teristics of attributes, For example, the value of 


(CHAR TIME FEVER) 


is the time at which the attribute fever was observed, 


By specializing the function BEFORE as 


BEFORE (TIME, RASH, TIME, FEVER, 2) 


The relationship for the disease in question can be checked, 

Such relationships are defined by the DEFINE subroutine which 
the user can invoke as required, Relationships can also be built 
into the information structure when it is first established if they 
are known to be necessary, To define a relattonship among the at- 
tributes of a particular state, one uses the CLUSTER routine, This 
routine re-organizes the state list for the state involved, producing 
an attribute-cluster. Thus, for the example above, the reorganized 
state list might look as that in Figure 6, As with individual at- 
tributes, a conditional probability given state is associated with 


each attribute cluster, 
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Figure 10 


State List with Cluster 


Any number of relationships can be defined for the structure 
provided that they can be expressed in the prescribed manner, Com- 
plex relationships can be specified by using functions of functions, 
Note that attributes remain independent for any state unless a re- 
lationship involving them is defined for that particular state, 
Thus, in one disease "fever" and "rash" may be related in some way, 
while in another they may be independent, 

The diagnostic program employs an interpreter to determine the 
truth of relationships during diagnosis, The interpreter permits 
the correct incorporation of relationships in the diagnostic infer- 
ence, The manner in which the interpreter is employed will be ex- 


amined in detail later, 


B, THE DIAGNOSTIC PROGRAM 

The diagnostic program and its associated routines are the 
heart of the system, These programs embody the various diagnostic 
strategies employed by the system, When one uses the system in 
the solution of a diagnostic problem, he interacts with the diag- 
nostic program, This program provides the interface between the 
user and the facilities of the system, There are three basic 
functions performed by the diagnostic program, (Although, in fact, 
each of these functions is delegated to a set of subroutines, it 
is convenient to consider them as logical functions of the diagnos- 
tic program.) In brief these three functions are: 


7 
1) The interpretation of the attributes of a particular 
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problem based on the information contained in the in- 
formation structure, This function is called the 
inference function. 

2) The selection of tests for the user to apply to the 

system being diagnosed in order to obtain further 
clues as to the system state, This is the test se- 
lection function, 

3) The analysis of the attributes of a problem to de- 

termine whether there are irrelevant attributes 

present or to detect attribute patterns from more 

than one system state occurring simultaneously, 

This is the pattern-sorting function, 
The design of the diagnostic program permits the alteration or 
replacement of any of these three functions independently of any of 
the others, This flexibility is important, because these functions 
are fundamental to this scheme for diagnosis, and it is necessary 
to study different versions of the functions, The possibility of 
changing individual functions without changing the remainder of the 
program greatly facilitates this study, 

Before the diagnostic program can be used in a particular 
problem area, an information structure for that area must be es- 
tablished, This requires that a disk file containing all the rele- 
vant information be created, The disk file can be created using 


the standard input and editing facilities of the time-sharing, The 
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formatting of the file, although specified, is quite simple, and if 
the necessary information is available, the only difficulty in 


creating the file is dealing with the large amount of information 
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which may be required, The information in the file consists of state- 


attribute relationships and test cost data. An example of a portion 
of such an input file is shown in Appendix 1, A system program 
processes the input file and from it constructs the information. 
structure for the problem area, 

A second file containing the loss structure for the problem 
area is required by the diagnostic program, At present this loss 
structure is always a matrix, Any element of this matrix, liy is 
the estimate of the loss for diagnosing seas j as state i. The 
exact manner in which this information is employed will be made 
clear below, 

As a preface to the discussion of the logical functions of 
the diagnostic program, consider this example of a particular 
application of the program, Suppose the program currently is set 
up to diagnose a certain group of diseases, This means that the 
appropriate information structure and loss structure have been es- 
tablished. A user wishing to invoke the assistance of the program 
does so by providing an initial problem statement, This statement 
is essentially a list of the attributes which have been observed, 
Assume for the example that this list is . 

* temperature of 102° 
* severe coughing 


* sore right ankle 
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As indicated in Figurell, the program first invokes the pattern 
sorting function for the current attributes, In this case, the 
pattern sorting function hypothesizes that the attribute "sore right 
ankle" is not relevant to the principal medical problem of the patient, 
and so removes it from the list of attributes for later investigation, 
After the attributes have been processed by the pattern sorting func- 
tion, the set of all diseases which exhibit the relevant attributes 
is obtained and a probability distribution for diseases piven these 
attributes and the "experience" in the information structure is 
created, The creation of this probability distribution is the task 
of the inference function, This distribution results from a considera- 
tion of both the current attributes and the knowledge of the various 
diseases, It is the current view of the diagnostic problem assumed 
by the program, 

Now the program invokes the test selection function, The 
object of this function is to select a good test for the user to 
apply to the patient in order to gain more information, In selecting 
this test, the test selection function considers the current proba- 
bilities of the various diseases, the cost of each test, and the 
usefulness of the results expected from the test. The user is in- 
formed of the test which has been selected, The test may be as 
simple as asking the patient questions about his recent exposure to 
other sick persons, or it may be more involved, for example, a chest 


X-ray, In any event, when the user has obtained the results of the 
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test, he reports them to the program. These test results are new 
attributes and the program again enters the loop shown in Figure 1l. 
This dialogue with the user continues until a diagnosis has been ob- 
tained, A more detailed trace of a session with the diagnostic 
program is presented in Appendix 2, This brief example provides an 
overview of the operation of the diagnostic program, In what follows, 
each of the primary functions of the program will be discussed in 


detail. 


1, THE PATTERN-SORTING FUNCTION 

As explained in an earlier section, only those attributes sig~ 
nificant to the diagnosis of a particular state are associated with 
that state in the information structure, Thus the attribute "sore 
ankle" would not be associated with the disease tuberculosis in the 
information structure; this means that the name of the attribute list 
for the attribute "sore ankle" does not appear on the state list for 
the disease "tuberculosis", Similarly the member list of the attrib- 
ute list for "sore ankle" contains no entry for the state list of 
tuberculosis, If the name of a state list does not appear on the 
member list of a given attribute list, then the conditional probability 
of the attribute given the state is taken ta be zero by the program, 
As will be discussed in the following section, the particular method 
of deduction employed by the program (Bayes' rule) results in a zero 


posterior probability for the state given the attribute, For instance, 
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if in the course of a diagnosis in which tuberculosis was considered 
a possible cause of the attributes the attribute "sore ankle" were 
observed, the updated probability of tuberculosis would be zero, 
Since the program removes from current consideration any state with 
zero probability, this approach makes maximum use of each attribute 
to reduce the set of possible diagnoses, 

The problem encountered here is that while "sore ankle" is 
not an attribute of tuberculosis, one certainly can have tuberculosis 
and a sore ankle, This is but one example of the more general prob- 


lem of irrelevant or noise attributes, Unless special precautions 


are taken, such attributes can eliminate the actual state from con- 
sideration when processed by the inference function, A number of 
solutions to this problem are possible, 

One approach is to associate every attribute with every state, 
employing £ probabilities whenever an attribute is not considered 
relevant to the diagnosis of a particular state, As long as € is 
greater than zero, no state will be eliminated from consideration in 
the manner described above, The difficulty is that this method pre- 
vents the drastic reduction in the set of possible diagnoses which 
is necessary for efficient operation of the program, A second ap- 


proach is to employ the ¢ probabilities as above, but to eliminate 


Ithis probability might be taken to be the unconditional proba- 
bility of the attribute, Since this probability may be quite small, 
the problem discussed here could still be encountered, 
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from further consideration those states whose posterior probability 
falls below a fixed threshold. This method is unsatisfactory be- 
cause the posterior probabilities for the various states can undergo 
radical change as additional attributes are observed and employed 
by the inference function, Thus, there is no guarantee that a state 
with a very low probability in the early stages of the diagnosis will 
remain improbable with the observation of new attributes, This 
problem can be even more severe if the noise attributes are the first 
observed, In either event, the actual state may be removed from fur- 
ther consideration by this method, Another approach is to decide 
whether an attribute is relevant to the diagnosis or merely noise 
before it is processed by the inference routines, This is a very 
difficult task to accomplish given the particular model employed in 
diagnosis by the program, The model of the system being diagnosed 
consists principally of state-attribute relationships without any 
information about causal connections, Thus, the only way to evalu- 
ate the relevance of an attribute to the diagnosis is to consider 
some measure of its probability given the diagnosis to date, Since 
almost every measure of this kind depends on the current prior dis- 
tribution, which, in turn, depends on the observed attributes assumed 
to be relevant, a cyclical argument results. 

A second problem arises when attributes characteristic of two or 
more distinct states are observed, as in the case of an individual 


with more than one disease, This is more than a problem of simple 
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noise since the program must detect two or more patterns, Again 
the methods mentioned above are inadequate to cope with this problem, 

The solution to this problem which has been incorporated in 
the diagnostic program involves processing a number of attribute 
patterns in parallel during a diagnosis, A pattern is a subset of the 
set of observed attributes which has the following two properties: 

1) At least one state in the information structure exhibits all the 
attributes in the pattern with a non-zero probability and 2) The 
pattern is not a subset of any other pattern, If the set of observed 
attributes contained a number of the attributes of tuberculosis and 
the attribute sore ankle, one pattern would be the set of tuberculosis 
attributes, A second pattern would be obtained by choosing a disease 
for which sore ankle is an attribute and taking the intersection of 
the set of attributes for that disease and the set of observed at-~ 
tributes, Perhaps the set of attributes obtained in this way, using 
a second disease on the member list of “sore ankle," might be dif- 
ferent from both those previously obtained, If so, this set is still 
another pattern, 

Throughout the course of a diagnosis, a pattern stack is main- 
tained by the pattern-sorting function, A schematic of the pattern 
stack is presented in Figure 12, Each pattern is represented by a 
sublist of the pattern stack, and associated with each pattern is 
the probability distribution for the states of the system given the 


attributes of the pattern, 
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Figure 12 


Pattern Stack 
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Whenever a new attribute is obtained in a diagnosis, it is 
processed against every pattern in the pattern stack, The new at- 
tribute is used to update a pattern if it is relevant to at least 
one state in the probability distribution for the pattern, After 
this updating, the attribute is added to the pattern, If no state 
in the probability distribution of a pattern is known to exhibit the 
new attribute, no changes are made to either the pattern or the dis- 
tribution, The actual manner in which distributions are updated to 
account for new attributes is discussed in detail in the next sec- 
tion on the inference function, 

When the new attribute has been processed against all patterns, 
a routine called PATFRM is invoked to form new patterns if possible. 
PATFRM retrieves the member list of the attribute list corresponding 
to the new attribute, For each state on the member list, the set of 
probability distributions in the pattern stack is searched. If the 
state is found in this set, the pattern for the state is already in 
the pattern stack, If the state is not found, the intersection of 
the set of attributes denoted by the appropriate state list and the 
set of observed attributes is a new pattern, This pattern and the 
corresponding distribution for the states is added to the pattern 
stack, While it is conceivable that this procedure could generate 
many patterns for a given information structure and attribute se- 
quence, this is not a serious problem. First in most areas the num- 


ber of distinct patterns which can be formed by this procedure for a 
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given attribute set is quite limited, because states exist in 
groups which have overlapping attribute patterns, Secondly, the 
number of patterns considered can be limited by considering only 
those patterns with a probability greater than some threshold, 

This procedure also includes a provision for removing patterns 
from the stack, If the inference function determines that the 
probability of a particular pattern is zero, the pattern and its 
associated distribution is eliminated from the pattern stack. The 
contents of the pattern stack, then, can be quite dynamic during 
the course of a diagnosis as new attributes trigger the addition 
and deletion of patterns, 

As an illustration of this aspect of the pattern sorting 
function, consider the following example, At a given stage in a 
diagnosis of a medical problem, three attributes have been observed. 
These attributes are A, B and C, Also assume that of the diseases 
represented in the information structure, none exhibits all three 
of these attributes, A number of diseases exhibit A and B as at- 
tributes, however, and so this is a pattern, The point here is that 
while a disease which exhibits A and B can occur with C also present, 
C is not considered relevant to the diagnosis of any of these 
diseases. For the diseases for which C is a relevant attribute 
A is also relevant. For this situation the pattern stack can be 
represented as in Figure 13A, Here the symbol 77 denotes the dis- 


tribution list for a pattern, 
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Now when the new attribute D is observed, it is processed 
through the pattern stack, Assuming that the new attribute is rele- 
vant to some of the states in distribution J[,, this distribution is 
updated by the inference function to produce WW and the attribute D 
is added to the pattern, Attribute D is not relevant to the second 
pattern in the stack, and so this pattern and its associated dis- 
tribution remain unchanged, Finally, the routine PATFRM is invoked 
to search for new patterns, Assume that no new patterns are formed, 
Thus, at the end of this phase in the processing of the new attrib- 
ute, the pattern stack appears as in Figure 13B, 

Now in the event that there is more than one pattern in the 
stack, the diagnostic program must make a decision as to which 
pattern to diagnose. Thus, the program must generate a hypothesis 
about the significance of the various patterns in the stack, For 
example, if one pattern corresponds to a majority of the attributes 
of tuberculosis, and the other to a single attribute "sore ankle," 
it is extremely important for the program to give priority to the 
former pattern, The problem is to establish pattern selection rules 
which will make the "correct" decision in such a case, 

One consideration which is relevant to the selection of a pat- 
tern is the seriousness of the states suggested by the pattern, For 
this reason, an attribute quite specific to a very serious disease 
will strongly influence the course of a medical diagnosis, 


In order to account for the relative seriousness of different 
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and TT ; = a priori probability of state j. 


Values of 9 decrease with increasing seriousness of states, This 


can be seen in the following simple example. 


LOSS 
See Ce eee o 
1, Benign tumor 0.7 0 1,000 , 000 300,000 
2, Malignant tumor 0.3 100 0 70 


While other more sophisticated measures of seriousness can be de- 
veloped, this simple one was deemed suitable for the purposes of 
this research, 

Once the seriousness of the various states has been estab- 
lished, the problem of pattern selection can be solved in a quite 
reasonable way through the use of the Bayesian model, For each 
pattern, a conditional distribution on states can be obtained by 
the inference function, For each pattern, the distribution is con- 
ditioned on the attributes of that pattern alone--all other patterns 


are ignored, Thus for the kth pattern 
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Where 1S is the conditional probability of the ;2 


state (M5) given the pattern f S4y eiveen% Sn}. 


The seriousness measure for the ge pattern is given by 
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and the pattern selected is the one with minimum y. 

This measure has several desirable properties, Consider the 
case of an attribute which is very specific to a serious disease, 

If that attribute is observed, the conditional probability for the 
serious disease given the pattern containing the attribute will be 
close to one, Since the corresponding value of 9 is small, the 
value of Y' for the pattern will be small, Hence this pattern will 
quite likely be selected, On the other hand, if the attribute is 
not specific to the serious disease, the conditional probability 
for the disease given the pattern will be less; and the resulting 
value of be greater. 

The measure also favors a pattern which contains many attributes 
provided that the pattern strongly indicated one or more serious 
states, The posterior distribution does not have to be spiked, how- 
ever, for a pattern to be chosen, For example a pattern which re- 
sults in equal probabilities for six states may also be chosen if 
the seriousness of the individual states so warrants, This measure 
accounts for both the specificity of a pattern and the seriousness 
of states associated with the pattern, In this respect, it seems to 
be a good way to select patterns for investigation, 

A routine called SELECT chooses the current pattern for the 
diagnostic program, and this pattern may change from time to time 
as additional information is gathered by the program, The current 


pattern. is the one employed by the test selection function for 
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evaluating tests. Before each use of the test selection function, 
SELECT chooses the current pattern based on all information cur- 
rently available. 

A number of other processing routines affect the pattern stack 
during the course of a diagnosis, Recall that whenever the pattern 
sorting function produces more than one pattern in the stack, the 
selection of a pattern for further diagnosis constitutes a hypothe- 
sis about the significance of a group of attributes, If a consist- 
ent diagnosis for the current pattern is obtained, then the hypothe- 
sis is tentatively confirmed, If there are no other attributes to 
account for then a consistent dineiseis for all attributes has been 
obtained, Otherwise the remaining patterns must be considered, It 
is possible that a second pattern is being diagnosed, new attributes 
may prove the hypothesis about the first pattern to be incorrect, 

In this case, the attributes in this pattern can no longer be con- 
sidered accounted for, These possibilities are dealt with in the 
following way by the pattern sorting function, The program maintains 
a list called the "unaccounted-for" list, and on it are all those 
attributes which have yet to be attributed to a particular system 
state, When the current pattern is "diagnosed" or assigned to one 
state, the attibutes in the pattern are removed from the unaccounted- 
for list, and the pattern itself is marked. A marked pattern is ig- 
nored in test evaluation, although it is updated with new attributes 
whenever appropriate, When the current pattern has been marked, all 


unmarked patterns are deleted from the stack, Then PATFRM is called 
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for each attribute in the unaccounted-for list. Patterns are formed 
using the unaccounted-for list as the total attribute set. If the 
unaccounted-for list is empty, a consistent diagnosis for all attrib- 
utes has been obtained, Otherwise, the diagnosis continues on the new 
patterns, 

This means that attributes which are included in marked patterns 
are not utilized in the formation of new patterns at this time. If, 
for example, the total attribute set were (A, B, C, D) and (A, B, D) 
had been tentatively diagnosed, the only unmarked pattern would be 
(C), This is true even though there may be states which exhibit both 
C and A, If, however, the test selection function chooses a test 
which can detect A, A will be added to the unmarked pattern, This is 
because the program always consults the history of the diagnosis be- 
fore requesting the user to run a test. If on the other hand, the 
program would normally account for C without employing knowledge of 
A, it will do so, 

If a new attribute causes the probability of a marked pattern to 
become zero, a special recovery procedure is invoked, First, each 
attribute of the marked pattern is transferred to the unaccounted-for 
list. If one of these attributes is added to the list, it is also 
processed against all the other patterns in the stack, When the stack 
has been updated with such an attribute, PATFRM is invoked to check 
for new patterns based on this attribute. Finally, the marked pattern 


is deleted from the pattern stack, and diagnosis continued, 
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Thus, the contents of the pattern stack may be quite volatile 
during a diagnosis, although cases of extreme volatility are not ex- 
pected to occur very often, In any event, the use of the pattern 
stack permits the program to deal with noise and multiple patterns 
in a reasonably efficient manner, By allowing the user to inter- 
act with the program during diagnosis, it is possible to employ his 


judgment with regard to the merits of pursuing particular patterns, 


2, THE INFERENCE FUNCTION 

In general, the observation of a new attribute provides the 
diagnostic program with additional information about the current 
state of the system being diagnosed. Based on this observation, the 
program may significantly alter its estimate of the likelihoods of 
the various states, This section discusses in detail the manner in 
which the program incorporates observations of attributes into its 
current view of the diagnostic problem. The routines which process 
new attributes for their effect on the current view of the problem 
collectively are called the inference function, 

The basic analysis of attributes and inference done by the diag- 
nostic program is based on Bayes rule. Bayes rule can be stated as 
follows 


_ PCM. /€ )P(S [M;.€) 
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where P(M, |Z) is the probability that the current state is M, 
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conditional on the total experience to date, 

P(S,/M;,£ ) is the probability that the system will exhibit 

attribute S, given that it is in state M; and the diagnos- 

tic experience z=, 

P(S./¢) is the probability of the system exhibiting S., 

unconditional on state, 

P(M;/S,,€ ) is the conditional probability that the state of 

the system is M; given @ and the newly observed attribute S,. 
The quantity P(M;/<) is called the prior probability and P(M;/S,,€ ) 
is called the posterior probability of the state M;, The observa- 
tion of the attribute S, increases the experience or information 
available on which to make a decision about the unknown state. The 
posterior probability is an adjustment of the prior probability to 
account for the new information, After this adjustment has been made, 
the posterior probability is the new prior probability when further 
attributes are observed, Consider the following example of this 
basic inferential process; 

Suppose there are only two states relevant to the current diag- 
nostic problem, Mj and M2, and three attributes 5,, Sj and S3, The 
a priori probabilities for the two states as well as the conditional 
probabilities for the attributes given the states are presented in 


Table 2, 
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TABLE 2 


EXAMPLE FOR BAYESIAN ANALYSIS 


Conditional 
Probability of Attribute/ State 
A priori 
probability 8) So 83 
My; 0.8 ol 4 el 
Mo 0.2 7 6 9 


The initial experience of the program, before any attributes have been 
observed, is embodied in the a priori probabilities, Thus, the cur- 
rent distribution on states is (0.8, 0.2). Now assume that tests 
employed in the diagnosis reveal the presence of attribute S,. Ac- 
cording to Bayes rule, the posterior distribution is (.82, .18). 


That is 
0,8)(0.8 
P(My/S1,€ ) = %ray(0,80)+(0.2)(0.7) = 0-82 


P(M,/S,,€) = ———-(0.2)(0,7) ss = (0,18 
/SL.€) (0.8) (0.8)+(0.2) (0.7) 


Thus, the new attribute has little effect on the view of the problem 
taken by the program, If two more tests yield the attribute S4 and 


then the attribute $3, the corresponding distributions are: 


P(M, /S,,8),€) = 0.75 — POM, [S1,82,€ ) = 0,25 
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tern and its distribution list are removed from the stack, While 
Bayes rule is easily applied in principle, the inference function 
must include special routines to insure that inter-attribute rela- 
tionships and the "history" of the diagnosis are correctly accounted 
for in the probabilistic analysis, 

The routine UPD which performs the updating of the pattern stack 
based on the observation of a new attribute is to a large extent a 
simple encoding of Bayes rule. The routine, however, does not ob- 
tain the requisite conditional probabilities directly. Instead, it 
calls PIJ to obtain the conditional probability of attribute "j" 
given state "i" and the history of the diagnosis to date, The reason 
for this indirection in the accessing of probabilities is really a 
pragmatic one, The insulation UPD from the probability-retrieving 
process allows changes in this process to be made without affecting 
the basic inference process, 

As noted, the function of PIJ is to retrieve conditional proba- 
bilities from the information structure, In the simplest case, this 
involves retrieving a number directly from the information structure, 
When the attribute of interest is involved in an attribute cluster for 
the given state, the process of determining the conditional probability 
is more involved, 

The general form of an attribute cluster is either 

a. (8, R,) 


or b. (8, Ry % 8, R, ?...% eo. R) 
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where R, is an inter-attribute relationship; 

9, is the conditional probability of R, given the state 

® is either "exclusive or" or "or." 
Here Rj can be any inter-attribute relationships (including functions 
of functions, etc.) as long as it does not include @, The reason for 
this restriction is to eliminate ambiguity from the probability 
assignments, In fact, the restriction does not limit the class of 
logical relationships which can be defined, only the form which in- 
dividual members may assume, Thus, for example, Rj might be the 
cluster for the relationship 

"Either A, precedes Ay in time or A, does not appear at all," 

In order to evaluate the conditional probability of an attribute 
involved in an attribute cluster, PIJ must be able to evaluate the 
truth of the relationships Rj- It does this by calling the routine 
INTERP to determine the true value of each Ry. INTERP is an inter- 
preter, which retrieves the definitions of any functions involved in 
Rj and applies these definitions to the appropriate arguments from 
the attribute cluster, The interpreter employs a push-down stack and 
recursive calls in the evaluation, All functions are reduced in this 
way to their component primitive functions, Routines to evaluate the 
primitive functions are built into the system. 

The operation of the interpreter differs in certain aspects 


from that of a normal interpreter of Boolean functions, because this 


interpreter must deal with variables whose current value is unknown, 
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For example, suppose the relationship under consideration for a par- 
ticular state M; is "A, precedes Ay in time" with probability 0.5. 
Assume A, has just been observed and the conditional probability of 
A, given the state is desired. If Ay has not yet been observed, 

the relationship is incomplete (or from a logical standpoint, unde- 
fined). From a Bayesian point of view, however, the conditional 
probability is well-defined; it can be obtained by assuming that A) 
will in fact follow A, in time, This assumption results in a value 


of 0,5 for the conditional probability of A, given M If Ag is ob- 


jc 
served later, then its conditional probability can be obtained in a 
similar manner, but the prior observation of A must be taken into 
account, This means that the desired probability of A, is conditional 


on the state M. and the previously observed A Hence the proper con- 


1° 
ditional probability is 1.0, 

In general terms, the interpreter assumes the truth of any 
relationship which is incomplete unless that relationship is demon- 
strably false given the current information of the diagnosis, The 
interpreter must also indicate whether any attributes involved ina 
cluster have actually been observed, Given these modifications of the 
interpreter function, the routine PIJ can deduce the proper conditional 
probability for the given attribute-state pair. PIJ embodies a number 
of logical tests on the truth of the Be and the number of observed 


attributes involved in each, For the types of relationships allowed 


in the information structure, these quantities are sufficient to deter- 
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the relevant attributes are first presented by the user, Through 
the use of the interpreter, the diagnostic program is able to deal 


with variety of relationships within a particular problem area. 


3, THE TEST SELECTION FUNCTION 

The value of heuristics for test selection in diagnostic prob- 
lems has been underscored in previous sections, In this section, a 
particular test selection program is discussed, This program (which 
is, in fact, a number of subroutines) is the one employed in the diag- 
nostic program, The nature of the program strategy and organization 
is explained and some of its limitations are noted, 

From the model of a diagnostic problem discussed in Chapter 3, 
it will be recalled that one of the major tasks in diagnosis is the 
selection of a good set of tests to apply to the system, The de- 
termination of such a testing strategy involves a consideration of 
both the costs of tests and the information which they are expected to 
yield, Thus, any heuristic for the test selection process should re- 
flect these considerations, Another consideration involves the amount 
of computation involved in applying the heuristic in a particular 
diagnosis, In order to facilitate the study of a class of such test 
selection heuristics, the test selection function was designed to be 
in large part independent of the particular heuristics employed, While 
the class of heuristics permitted is not particularly large, it does 


include heuristics which lead to markedly different test selection 
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searched by the test selection function during a particular stage in 
a diagnosis are searched to the same depth, ! The limitations arising 
from this inflexibility will be discussed later, 

The breadth of the search is controlled indirectly by the user 
through the use of a threshold probability. At a given decision node, 
only those tests which are relevant to a state with a probability 
greater than the threshold are considered by the test selection func- 
tion, For example, if the probability distribution at a given decision 
node is (0.2, 0,3, 0.5) for states M,, M,, M, and the threshold is 0,25, 
only those tests relevant to states My and to M3 will be considered, 
Those tests which are relevant to M, alone will be ignored, A test 
is considered relevant to a particular state only if an attribute 
which is associated with the appropriate state list in the information 
structure is a possible result of the test given the probability dis- 
tribution for the current decision node, Since the control of the 
breadth of search is indirect, in general, the user cannot easily 
predict the extent of the pruning of the decision tree which will 
result, Some feeling for reduction in the search space can be gained 
from experience with the program in a particular problem area, Note 


that in the above example, if all the tests which are relevant to 


An exception occurs when a particular node corresponds to a 
certain diagnosis, The search of the branch containing this node 
will terminate there, 
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state M; are also relevant to either Mj or M3, then the threshold 
probability will not result in any pruning of the decision tree, 
The maximum search breadth is obtained with a threshold of zero, 

Like the search depth parameter, the threshold parameter can 
be set prior to each stage in the diagnosis, Also these two para- 
meters can be varies independently of one another (subject only to 
a practical constraint of available storage). This flexibility per- 
mits the overall selection strategy to change during the course of 
the diagnosis, 

There are four routines in the test selection package, each 
performing a distinct function in the tree search, The principal 
routine is SEQDEC which serves as the main control for the process 
of test selection. The diagnostic program communicates with the 
test selection package through SEQDEC, It provides this routine 
the name of the node in the decision tree which corresponds to the 
current state of the diagnosis, SEQDEC then analyzes the tree to 
the appropriate depth and breadth to obtain the testing decision, 

Because the decision tree can require considerable storage 
even for limited search depth and breadth, the tree is developed dy- 
namically, That is, new levels are added only as they are needed, 
and levels are erased when they have been analyzed, SEQDEC is called 
with the name of a decision node as an argument, This decision node 
is represented by an empty SLIP list which has on its DLIST a list 
containing a probability distribution over system states, This dis- 


tribution incorporates all the attributes which were observed on the 
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path from the beginning of the tree to the current node, 

SEQDEC first determines the expected loss for an optimal decision 
at this node, The manner in which this value is determined will be 
explained below. If the level of the current node equals the re- 
quired depth of search this expected loss is returned as the expected 
loss for the node. If not, the current loss for this node is assigned 
this value and if the level of the node is the topmost level of the 
analysis, the terminal decision and its value are stored in a special 
list. In any event an additional level must be "grown" on the tree, 
First the routine RELTST is called by SEQDEC, RELIST determines the 
set of tests which are relevant to the states whose probability at 
the current node exceeds the threshold, Excluded from this set are 
all those tests which have been actually run. These latter tests are 
known to RELTST because whenever a test is selected by the diagnostic 
program and run by the user, its name is placed on a list called 
TSTRUN in common storage, RELIST stores the names of the relevant 
tests on the current decision node list. 

After RELTST has collected the set of relevant tests, SEQDEC 
processes each of these tests in turn, SEQDEC begins reading the 
list of tests. For each test, a routine called GROW] is invoked, 
This routine determines all possible results of the given test and 
their respective probabilities, For each result, the routine con- 
structs a new decision node, First the current test is placed on the 


top of TSTRUN to simulate the running of the test and then for each 
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of the possible results of the test, SEQDEC calls itself recursively 
to obtain the expected loss of the resulting decision node, When 

this value has been obtained, it is weighted by the probability of 

the given result and the product accumulated, The sum of the expected 
loss for each result is combined with the cost of the test, The cur- 
rent test is removed from TSTRUN and the portion of the decision tree 
which has just been analyzed is erased, If the analysis is at the 
topmost level the value of the test is saved. This means that the 
expected losses for all alternatives at the current level are available, 
In the event that the best alternative cannot be employed (e.g. a test 
cannot be run for some reason), the next best alternative can be 
chosen, In any case, the expected loss for this test is compared with 
that of the best decision to date for the node, If it is less, the 
current test becomes the best decision, The analysis then proceeds 

to the next test alternative, When all alternatives have been evalu- 
ated for the current decision node, SEQDEC returns the expected loss 
of the best decision as determined by the analysis, 

The determination of the optimal terminal decision as accomplished 
by a routine called DLOSS. This routine employs the probability dis- 
tribution, the decision node and the loss function to determine the 
value of the minimum expected loss terminal decision for the node, 

T£ uP is the probability of the state M, in the current distribu- 
tion and 1; is a typical element from the loss function matrix, DLOSS 


selects state My where 
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and E is the expected loss of the optimal terminal decision for the 
node, The state selected by DLOSS and the value E are returned to 
SEQDEC, 

By controlling the breadth and the depth of the search employed 
by the test selection function, the user can generate a number of 
different test selection heuristics, For example, he might use a 
threshold close to zero and a depth of one early in a diagnosis 
when many states are still possible, Because the probability dis- 
tribution based on only a few attributes may be quite diffuse, a 
low threshold is needed to insure that significant tests are not 
overlooked. On the other hand, the potentially large number of de- 
cision nodes requires a limited depth of search. As the diagnosis 
progresses and a few states become relatively probable, the thres- 
hold can be raised with less danger of missing significant tests. 
With the higher threshold it may be possible to improve the evaluation 
of tests by increasing the depth of the search, 

The selection scheme above can be supplemented by the use of 
two additional features of the program, First, the user can re- 
strict the set of relevant tests to those associated with the best 
terminal decision at a given node, In the case when the loss function 
is a constant for all ordered pairs of states, this corresponds to 


considering the tests which are relevant to the most probable state, 
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Since the routine DLOSS can determine the best terminal decision at 
a given decision, the appropriate state can be made available to RELTST, 
By considering only the tests relevant to this state, the user in 
a sense in limiting the search to those tests which will tend to 
prove or disprove the hypothesis that the given state is indeed the 
best decision, In practice, the user obtains this option by setting 
the threshold probability to a number greater than one, 

In order to permit the user an even greater facility to test 
hypotheses, the program permits him to request a search for tests 
to prove or disprove the hypothesis that “the state of the system 
is M,." If the user chooses to test such a hypothesis, the test se- 
lection function will alter its method of evaluating decision nodes, 
All decision losses (1#j) are set temporarily to a certain very 
high value, The routine DLOSS then considers only two states in its 
evaluation of the loss for a given node, One state is M, and the 
other is "not M.." With these adjustments, the test selection func- 
tion will rank tests according to their expected value in proving or 
disproving the presence of state M. 

A comparison of a number of particular selection heuristics 


employed in this research will be presented later in the thesis. 


C, THE GENERATOR PROGRAM 
The diagnostic program discussed in the previous sections is a 
major tool in this research. By exploiting the interactive capabili- 


ties of the program, the user can employ it directly in the solution 
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of actual diagnostic problems. Of equal importance, however, is the 
availability of the program as a test vehicle for a variety of over- 
all diagnostic strategies, By specifying the heuristics to be em- 
ployed in the pattern sorting and test selection functions, one is 
defining a diagnostic strategy. Since diagnostic problems tend to 
be difficult and the program operation is quite complicated, it is 
not an easy task to make generalizations about a given diagnostic 
strategy. There are many important questions which can be asked about 
a diagnostic strategy such as 

* How is the performance of the program affected by noise signs? 

* What is the effect of uncertainty in the probabilities on 

the performance of the program? 
* How do various changes in the relevant probability distribu- 


tions affect program performance? 


Questions such as these are difficult to answer based on experience with 
only a few problem areas, If one is constrained to work with descrip- 
tions of actual systems, it may be very difficult to establish the 
conditions required for the test of a particular aspect of the pro- 
gram, If, on the other hand, one can employ a wide variety of system 
descriptions, the program can be exercised more thoroughly, One ap- 
proach is to create an information structure with the desired proper- 
ties and to test the diagnostic program with simulated problems from 
this artificial problem area, Information gained from such studies of 


diagnosis "in the abstract" may suggest improvements in the program. 


105 


It may also provide a deeper insight into the problems involved in 
solving real diagnostic problems, If such a simulation facility 
were available, simulated cases generated from the structure for an 
actual problem area could be utilized to conveniently investigate 
aspects of diagnosis in that area, 

The diagnostic system includes such a simulation facility in 
the form of the generator program, This program is the third major 
part of the diagnostic system, Like the diagnostic program, the 
generator makes extensive use of the information structure, The 
system for which problems are to be simulated is described in the 
standard manner by the user, This description is converted to an 
information structure which is available to both the diagnostic pro- 
gram and the generator, The basic operation of the generator is as 
follows, First, a state is chosen at random from the set of possible 
states for the system in accordance with the a priori probability 
distribution, Then a certain number of initial attributes (the 
number being controlled by the user) are generated at random given 
the description of the state in the information structure, The set 
of initial attributes constitutes the problem presented to the diag- 
nostic program, The latter is called to process these attributes, It 
selects a test in the usual manner, Given the state and the test, 
the generator selects a test result and conveys this response to 
the diagnostic program, This interaction between the generator and 


the diagnostic program continues until the latter arrives at a diagno- 
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sis, This diagnosis then can be compared with the "known" state 
used by the generator. 

As an example of the operation of the generator, consider its 
use in the following simplified problem, The generator is used 
to simulate disease case histories for the disease-attribute proba- 
bility matrix presented in Table III, The relevant tests are listed 
to the right of the matrix, Assume that cases are to be drawn at 
random from the structure and that one initial attribute is to be 
presented to the diagnostic program, 

The generator first selects the disease, It does this by creat- 
ing a list of all possible diseases and cummulative probabilities, 


For this example, the list would be 
(D1 0,3 D2 1.0) 


Each cummulative probability is the sum of the a priori probabili- 
ties of the diseases preceding it in the list. Then a random number 
between zero and one is generated, The list of diseases and cummula- 
tive probabilities, called the generation list, is searched for a 
disease with the property that the probability preceding it is less 
than and the probability following is greater than the given random 
number, This disease satisfying this condition is chosen for this 
case, Thus, if the random number generated in the example were 0,41, 
the disease selected would be D2. Assuming the disease D2 has been 


chosen, the generator now selects the initial attributes which define 


TABLE 3 


Disease Description for Generator Example 


a_priori 


Disease Probability 


D1 0.3 


D2 0.7 


Test 
Tl 
T2 
T3 


T4 
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P(Attribute/Disease) 


AL A2 AS ASA 
0.3 0.7 0.5 1,0 0.5 


0.8 0.2 O.3 0.2 0.6 


Attributes 


Al, A2 


A3 
A4 


A5, A6 


AG 
0.5 


0.4 
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with the appropriate probabilities and returns it to the diagnostic 
program, This iterative process continues until the diagnostic pro- 
gram has completed the diagnosis, 

In this example, only one attribute was generated for each test, 
There are tests, however, from which several attributes can be ob- 
tained, Such tests are marked in the information structure, and the 
generator will generate a set of test results for these tests, 

The diagnostic system will record an extensive history of each 
diagnosis or selected aspects of that history on a history file if 
requested to do so by the user. A schematic of the relationships 
among the three major parts of the diagnostic system is presented 
in Figure 14. In the remainder of this section, certain features of 
the generator-diagnostic program interaction will be discussed in 
detail, 

The subroutine GETSYM is the principal link between the genera- 
tor and the diagnostic program, It is this routine which is called 
by the diagnostic program whenever the latter requires a test to be 
run, If the diagnostic program is being controlled by the user from 
the console, then GETSYM retrieves the test results from him, If 
the generator is in control, a routine called GENSYM is invoked to 
generate an appropriate response to the chosen test. The diagnostic 
program itself is independent of the source of responses to tests, 
GENSYM is also used by the generator to select the initial attributes 
of a problem, All system output (such as requests for test results, 


distributions, etc.) is processed by a special output package, This 
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Figure 14 


Schematic of Diagnostic System 


Ill 


another advantage is that many cases can be simulated in a rea- 


sonable amount of time. 


The second problem encountered in the diagnosis of bone 
tumors is the large number of potentially useful attributes which 
can be extracted from a radiograph, Generally speaking, there are 
four direct kinds of information which are obtained from a radio- 
graph of a bone tumor (R-20) 
1) Destruction of bone 
2) Proliferation of bone 
3) Mineralization of tumor matrix 
4) Location, size, and shape of tumor, 
Each of these general classes of information is broken down into 
a number of more specific attributes, The result is the large num- 
ber of attributes mentioned above, Hence, the diagnostician is con- 
fronted with a considerable amount of data which he may employ in 
classifying a particular tumor, 
The particular study discussed here involved the diagnosis 
of actual cases of bone tumors, each of which was classified into 
one of nine histological types, These types are listed in Table 4, 
The evidence employed in the diagnoses consisted of fifty-three 
attributes obtained principally from radiographs, (The age of the 
patient was the only non-radiologic attribute considered.) The 
attributes are listed in Table 5 along with their abbreviations 
used in discussions of particular diagnoses, 
The case histories and the disease-attribute probability matrix 


used in this study were obtained from Dr. G. S. Lodwick of the 
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University of Missouri, Dr. Lodwick and his associates developed 

the matrix as a result of many years experience with cases of bone 
tumors, Thus, the matrix represents the distillation of extensive 
diagnostic experience with the problem, It reflects both the stat- 
istical experience and understanding of the disease processes in- 
volved of the workers who created it, The papers cited above sum- 
marize their work and are recommended to any reader who is interested 
in a more authoritative view of the problem than the competence of 


this author permits him to present. 


B, Experiments in Bone Tumor Diagnosis 


The diagnostic system was used to study various aspects of 
bone tumor diagnoses, The disease-attribute probability matrix pro- 
vided by Dr, Lodwick was used as the basis for an information struc- 
ture for the system. A state was defined for each of the nine types 
of bone tumor, A set of thirty-two tests were defined, Some of 
these tests such as that of determining the age of the patient can 
result in one of a number of attributes, In the case of the age 
test, the possible attributes are: 1) age 0 to 9 years, 2) age 10 
to 19 years, 3) age 20 to 29 years, 4) age 30 to 39 years, and 
5) age 40 years and over. Other tests are specific for one attrib- 
ute, such as the test of checking for geographic destruction of 
bone, The set of tests and the respective attributes which may re- 


sult is presented in Table 6. Throughout the remainder of this 


TABLE 4 


HISTOLOGICAL TYPES FOR BONE TUMOR DIAGNOSIS 


Type Abbreviation 
1. Chrondoblastoma cB 
2. Chrondosarcoma cs 
3. Ewing's Sarcom ES 
4, Fibrosarcoma Fs 
5. Giant Cell Tumor Gc 
6. Osteosarcoma Os 
7. Parosteal Sarcoma PS 
8. Reticutum Cell Sarcoma RC 
9, Chrondomyzoid Fibroma CF 


Relative 
Incidence: 


0.05 
0.17 
0.15 
0.10 
0.15 
0.25 
0,05 
‘0.05 
0.03 


1.00 


Note: This formulation assumes that each patient has one and 


only one of the given diseases. 
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TABLE 5 


ATTRIBUTES FOR BONE TUMOR DIAGNOSIS 


Attribute Meaning 
S02 Age 00-09 years 
S03 Age 10-19 years 
S04 Age 20-29 years 
SO5 Age 30-39 years 
S06 Age 40 years and over 
SO7 Tumor Size 01-30 Millimeters 
S08 Tumor Size 31-60 Millimeters 
S09 Tumor Size 61-90 Millimeters 
S10 Tumor Size 91 MM and over 
$11 Shape-Round (L LT 1,5 X W) 
S12 Shape-Elongated (L GE 1.5 X W) 
$13  Location-Central 
$14 Location-Eccentric 
S15 Location-Cortex/Parosteal 
S16 Long Bone 
$17 Flat Bone 
$18 Small Bone 
$19 Sacrum and Pelvis 
820 Any Bone-Epiphysis 
$21 Any Bone-Growth Plate 
$22 Tubular Bone-Articular Cortex 
$23 Tubular Bone-Metaphysis 
$24 Tubular Bone~-Shaft 
$27 Matrix-Radiolucent 
$28 Matrix-Floccules 
$29 Matrix-Solid 
S30 Matrix-Lumpt 
$31 Matrix-Clouds 
$32 Destruction-Geographic 


Destruction-Motheaten 


Attribute Meaning 
834 Destruction-Permeated 
835 Margin-Regular 
$36 Margin-Lubulated 
837 Margin-Ragged 
$38 Margin-Indistinct 
S39 Transition Sharp or Smudged 
S40 Invaaive Zone 
841 Special Sign-Fracture 
S42 Special Sign-Displacement 
843 Proliferation-Sclerotic Rim 
844 Prolif.-Multiple Small Foci 
845 Proliferation-Endostosis 
S46 Periosteal-Hyperostosis 
S47 Periosteal-Buttress 
$48 Periosteal-Trabeculae (Septae) 
S49 Cortex Expanded 
S50 No Codman's Triangle 
S51 One Codman's Triangle 
S52 Two or More Codman's Triangles 
$53 No periostosis 
$54 Laminated Periostosis 
$55 Amorphous Periostosis 
$56 No Spiculation 
$57 Sunburst Spiculation 
$58 Hair-on-end Spiculation 
$59 Velvet Spiculation 
S60 Periosteal Response-Delicate 
S61 Periosteal Response-Coarse 
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TABLE 6 


TESTS FOR BONE TUMOR DIAGNOSIS 


Test Possible Results 
1, TEST2 S02, S03, S04, S05, S06 
2. TEST7 $07, S08, SO9, S10 
3, TEST11 S11, S12 
4, TEST13 $13, 814, S15 
5, TEST16 $16, S17, S18, S19 
6. TEST20 S20, N 
7, TEST21 $21, N 
8. TEST22 $22, N 
9, TEST23 $23, N 
10, TEST24 $24, N 
11, TEST27 $27, N 
12, TEST28 $28, N 
13. TEST29 $29, N 
14, TEST20 $30, N 
15, TEST31 S31, N 
16. TEST32 $32, N 
17, TEST33 $33, N 
18, TEST34 $34, N 
19, TEST35 $35, S36, S37, S38 
20, TEST39 $39, S40, N 
21, TEST41 S41, S42, N 
22, TEST43 843, N 
23, TEST44 S44, N 
24, TEST45 845, N 
25, TEST46 S46, N 
26, TEST47 S47, N 
27, TEST48 S48, N 
28, TEST49 S49, N 
29, TESTS50O $50, $51, S52 
30, TEST53 $53, 554, 855 
31, TEST56 $56, S57, S58, S59 
32, TEST60 S60, S61 


Note: The symbol "N" denotes a "normal" attribute, It means 
that a test may fail to reveal any of the other attrib- 
utes listed, Thus, for TEST41, the possible results are 
S41 or S42 or neither S41 nor S47 (N). 
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chapter, the abbreviations for diseases and attributes presented 

in Table 5 and Table 6 will be used. In the initial set of ex- 
periments, all tests were assigned unit cost and the cost of all 
misdiagnoses (e.g, deciding the tumor is CS when it is really GC) 
was assumed to be 100,000. This number is quite arbitrary, and is 
used simply to make the decision losses much greater than the test- 


ing losses, 


Experiment 1. Diagnosis Based on All Attributes 

Each of the twelve case histories was presented to the diag- 
nostic program by inputting all the attributes for the case, The 
diagnostic program processed the attributes through the inference 
function and obtained a posterior distribution for the type of 
tumor, The results of this experiment are presented in Table 7 along 
with the diagnosis of a pathologist provided with each case history. 
The latter is traditionally accepted as the definitive diagnosis 


in cases of this type. 


Experiment 2, Sequential Diagnoses--Actual Case Histories 

The second experiment exercised the sequential capabilities 
of the diagnostic program. Again, all diseases were taken to be 
equally serious Gy, = 100,000, i # j) and all tests were assigned 
unit cost. The same twelve cases were analyzed by the program, 
For each case, the program was presented with a set of initial at- 


tributes, This set was obtained by collecting the results of the 


119 


TABLE 7 


Diagnoses Based on all Available 


Attributes for Actual Bone Tumor Case Histories 


Case Posterior Distribution™ Pathology 

1 CB 0,12 GC 
Gc 0.87 

2 OS 0.65 os 
cs 0.35 

3 cB 1,00 CB 

4 cs 0.99 cs 

5 os 1,00 os 

6 ES 33 RC 
RC 67 

7 cs 0.78 © cs 
FS 0,22 

8 ES 0.04 ES 
ES 0,02 
RC 0.94 

9 ES 1.00 ES 

10 cs 1,00 CB 

11 GC 0.65 , GC 
CF 0.35 : 

12 PS 0.99 PS 


* Only types with posterior probability greater than or equal to 
0.01 are shown in the tables in this chapter, 


120 


first ten tests listed in Table 6 from the case histories, Thus each 
diagnostic problem was defined by approximately ten attributes, (In 
certain cases this number was smaller, because some tests are not 
relevant to specific bones.) 

After processing the initial attributes, for the case, the pro- 
gram employed the test selection function to select a test to be run, 
The results of the test selected were determined by consulting the 
given case history. The attribute or attributes resulting from this 
test were given to the program and the inference-test selection cycle 
repeated, Throughout this experiment the test selection function 
searched the decision tree to a depth of one and limited the breadth 
of search to those tests relevant to the most likely disease type. 

For each case, this sequential diagnosis was continued until 
the diagnostic program terminated the process. This termination 
occurred when the program determined the expected reduction in loss 
for the best test at the current decision node was less than the 
cost of the test, 

An example of a sequential diagnosis is presented in Table 8 and 
the results of the experiment are summarized in Table 9, 

The results of Experiment 2 underscore the potential advantage 
of sequential analysis of attributes in diagnosis, Since all diseases 
were taken to be equally serious for this experiment, the program 
found the best terminal decision to be the most probable disease, Since 
these same conditions held in Experiment 1, it is easy to make compari- 


sons between the results of the two experiments, 


Test 


TEST29 


TEST50 


TEST56 


TABLE 8 


Sequential Diagnosis--An Example 


(Actual Case History 12) 


Resulting Attributes 


S05, $10, $12, $15 
S16, NOT $20, NOT S21 
NOT S22, $23, S24 


$29 


$50 


$56 


Terminal decision -- PS 
Pathology report -- PS 


Posterior 


CS 
ES 
FS 
PS 
RC 


cs 
FS 
Os 
PS 


es) 
FS 
PS 


cs 
FS 
PS 


Distribution 


0.42 
0,13 
0.10 
0.31 
0,02 


0.06 
0.02 
0.01 
0.91 
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TABLE 9 
Sequential Diagnosis of Bone Tumor Cases 
Summary of Results for Actual Case Histories 
Number of Distribution at Distribution 
Case and Tests Selected Point of Terminal When all Attrib- 
Pathology by Program Decision utes Considered 
1. (GC) 9 CB 0,21 cB 0.12 
cc 0.78 Gc 0,87 
2. (0S) 12 cs 0,79 cs 0.65 
os 0.21 os 0.35 
3. (CB): 0 CB 1.00 cB 1.00 
4. (cS) 4 cs 0.80 cs 0.99 
ES 0,08 
FS 0,08 
Os 0.04 
5. (0S) 4 cs 0,03 os 1,00 
ES 0,02 
OSs 0.94 
RC 0.03 
6. (RC) 13 ES 0.30 ES 0,33 
FS 0.01 RC 0.67 
RC 0,68 
7. (CS) 4 cS 0,74 cs 0.78 
FS 0,26 FS 0,22 
8. (ES) 11 ES 0.05 ES 0.04 
Fs 0.07 FS 0.02 
RC 0,87 RC 0,94 
9, (ES) 5 cs 0,02 ES 1.00 
ES 0.88 
oS 0,05 
RC 0.05 
10. (CB) 3 CB 0,96 cB 1.00 
CF 0,04 
11, (GC) 5 cs 10 Gc 0,65 
ES m CF 0.35 
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12, (PS) 3 cs 0.05 PS 0,99 
FS 0,02 
PS 0,93 


Average number of initial attributes 9,4 
Average number of test by program 7,1 
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With regard to "accuracy," it can be seen that the lists of 
terminal decisions from the two experiments are identical and these 
decisions are the same as those of the pathologist in ten of the 
twelve cases, The major difference between the two sets of results 
is the average number of tests performed per diagnosis. In the first 
case this average is 30, (The average is less than 32 because some 
test results were not available or were not relevant for a given 
case and the test was not counted.) Sequential analysis of the given 
cases required an average of 16.7 tests per case, This average in- 
cludes 9.4 tests on the average to obtain the initia] attributes, 
Thus, by employing sequential analysis, the program in each case 
obtained the same diagnostic decision as it obtained using all attrib- 
utes, but with only slightly more than half as many tests, 

The nature of diagnosis of bone tumors makes this saving seem 
immaterial, That is, almost all attributes are obtained from a 
radiograph, and once the radiograph has been obtained, the marginal 
cost of the tests considered here is essentially zero. One can 
easily imagine a situation, however, in which tests are completely 
independent of one another. In such a situation, the savings from 
sequential diagnosis might be quite significant, The fact that the 
performance of a diagnostician should be assessed in terms of both 
accuracy and cost favors the sequential mode of operation for the 
program, The question of how to assess the performance of a diag- 


nostician will be considered at greater length later, 
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Another difference between the results of the two experiments 
is found in the posterior distributions at the points of a terminal 
decision, The average value of the maximum likelihood probability 
for the taruieal decisions can be taken as an indication of the 
equivocation or uncertainty in the average decision. For Experi- 
ment 1 this value is 0.85 while for Experiment 2, it is 0.80. 
Therefore, the sequential diagnoses terminate on slightly less 


"certain" decisions, 


Experiment 3. Sequential Analysis--Simulated Case Histories 

Table 10 presents the results of the sequential diagnoses of 
ten simulated case histories. The generator function was used to 
develop the cases and the diagnostic program employed as usual. 
Again, all diseases were taken to be equally serious and all tests 
were assigned unit cost, 

Again, the marked advantage of sequential diagnosis is evi- 
dent, The average number of tests required for diagnosis was 17.0, 
Based on a maximum likelihood terminal decision, the diagnostic pro- 
grams terminal decision was correct in nine of ten cases, 

On the average, the diagnostic program was more certain of its 
terminal decisions than in the previous experiments (average proba- 


bility of terminal decision = 90.5). 
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TABLE 10 
Sequential Diagnosis of Simulated Case Histories 
Distribution at 
Histological Number of Ini- Number of Tests Point of Terminal 
Type tial Attributes Selected by Program Decision 
1, FS 14 9 cS 0,26 
FS 0.73 
2, &ES 7 9 ES 0,88 
oS 0.01 
RC 0,11 
3, OS 11 0 OS 1.00 
4. GC 5 11 cs 0,01 
GC 0.79 
CF 0.20 
5. ES 12 6 cs 0,01 
ES 0.94 
OS 0.04 
6, RC 5 8 cs 0.05 
FS 0.78 
RC 0,16 
7. G 11 8 CB 0.93 
GC 0,02 
CF 0.05 
8. OS 11 8 Os 0,98 
cs 0,02 
9. FS 5 12 cs 0,11 
FS 0.88 
10, GC 10 8 cB 0.04 
Fs 0,01 
GC 0.94 
CF 0.01 
Average number of Average number of 
initial attributes tests by program 


9,1 7.9 


Chapter 6 


DIAGNUSIS OF CONGENITAL HEART DISEASE 


A, The Nature of the Diagnostic Problem 


A prolonged study of a group of thirty-four types of congeni- 
tal heart disease has been conducted by Warner and his associates 
(R12, R13, R14). As a result of this study, they developed a 
disease-attribute probability matrix for thirty-five types (includ- 
ing "normal") and fifty-seven attributes, The attributes can be 
grouped into four main categories: murmurs, electrocardiogram find- 
ings, X-ray findings, and other symptoms and physical signs. The 
problem of diagnosing heart disease cases based on this matrix is 
more difficult than the bone tumor problem discussed in Chapter 5. 
One reason for the increased difficulty is simply the increased 
number of diseases. Also certain groups of diseases have quite 
similar attribute probabilities in the matrix, 

As noted in Chapter 2, Warner developed a computer program to 
perform diagnosis of congenital heart disease patients based on a 
Bayesian analysis of their signs and symptoms, His program employs 
the matrix mentioned above, but in addition it must account for cer- 
tain dependencies (such as mutual exclusion of signs or symptoms), 
From the performance measures presented in Chapter 2, it can be 
seen that Warner's program performs at the level of an experienced 


physician, 
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The experiments discussed here involved the use of the disease- 
attribute probability matrix prepared by Warner in the diagnosis of 
congenital heart disease, As before, the matrix was the basis for 
each of the disease types and the appropriate attribute lists created, 
Twenty-eight tests were also defined for the problem, Dr. Warner 
provided nine case histories, each with the correct diagnosis and 
the diagnosis obtained by his program, In this instance, the cor- 
rect diagnoses were determined by follow-up studies such as heart 
catheterization or autopsy. 

Table 11 presents the names of the thirty-five states of the 
information structure used in these experiments and the names of 
the corresponding diseases. Table 12 lists the attributes of the 


problem; and Table 13 the tests, 


B, Experiments in Congenital Heart Disease Diagnosis 
Experiment 4, Diagnosis Based on All Attributes 


The first experiment tested the diagnostic capability of the 
program given all the known attributes for each of the actual case 
histories provided by Dr. Warner. The results of this experiment 
are summarized in Table 14, In each instance, the diagnostic pro- 
gram duplicated the results obtained by Warner's program for the 
given case history, (That is, both programs arrived at the same 


posterior probability distribution given all attributes.) 


States 


DOL 
DO2 
DO3 
DO4 


DOS 
DO6 


DO? 


DO8 


DO9 
D1O 


Dll 
D12 
D13 
D14 
DL5 


D116 
D17 


TABLE 


ll 
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Heart Disease Types 


Diseases 


Normal 

Atrial septal defect 

Atrial septal defect with 
pulmonary stenosis 

Atrial septal defect with 
pulmonary hypertension 
Atrio-ventricular communis 
Partial anomalous pulmonary 
venous connection 

Total anomalous pulmonary 
venous connection 

Tricuspid atresia 

(without transposition) 
Ebstein's anomaly 

Ventricular septal defect with 
valvular pulmonary stenosis 
Ventricular septal defect with 
infundibular pulmonary stenosis 
Pulmonary stenosis, valvular, 
gradient = 40 mm. Hg. 
Pulmonary stenosis, infundibu- 
lar, gradient = 40 mm. Hg. 
Pulmonary atresia 

Peripheral pulmonary stenosis 
Pulmonary hypertension 

Aortic pulmonary window 


States 


D18 
D19 
D20 
D21 
D22 


D23 
D24 


D25 
D26 
D27 
D28 
D29 
D30 
D3L 
D32 
D33 
D34 


D35 


Diseases 


Patent ductus arteriosus 
Pulmonary arterio~venous Fistula 
Congenital metral disease 
Primary myocardial disease 
Anomalous origin or coronary 
artery 

Congenital aortic disease 
Ventricular septal defect with 
pulmonary flow = 1.4 systemic 
flow 

Coarctation of aorta 

Truncus arteriosus 
Transposition 

Rypertrophic subaortic stenosis 
Absent aortic arch 

Ventricular septal defect with 
pulmonary flow > 1,4 systemic 
flow 

Ventricular septal defect with 
pulmonary hypertension 

Patent ductus arteriosus with 
pulmonary hypertension 
Tricuspid atresia with 
transplantation 

Pulmonary stenosis gradient 

<= 40 mm, Gh, 

Ruptured sinus Valsalva 
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Sign 


$01 
802 
$03 
S04 
S05 


S06 
S07 
S08 
so9 
S10 
$11 
812 
$13 
814 
$15 
S16 
$17 
$18 
) S19 
$20 
$21 
$22 
$23 
824 
$25 
$27 
$28 


se eg AR er I A I A A 


Age, 
Age, 
Age, 


TABLE 12 


Attributes for Congenital Heart Disease 


Meaning 


less than 1 year 
1 year to 20 years 
20 or more years 


Cyanosis, mild 
Cyanosis, severe (with 
clubbing 


Cyano 
Cyano 
Squat 
Apex 
Apex 
Apex 
Apex 
Apex 
Apex 
L 4th 
L 4th 


PAMPER rP rrr ree 
NS 
3 
i 


sis intermittent 
sis differential 
ting 
systolic 
systolic, holo 
systolic, mid 
diastolic 
diastolic, early 
diastolic, late 
systolic 
systolic, holo 
systolic, mid 
continuous 
diastolic 
diastolic, holo 
diastolic, early 
systolic 
systolic, holo 
systolic, mid 
cont inuous 
systolic 
diastolic 


Sign 


$29 
$30 
$31 
S35 
$36 
$37 
$38 
S40 


S41 
$42 
$43 


S44 
S45 


S46 
$47 
S48 
S49 
850 
$51 
852 
$53 
S54 
$55 
$56 
$57 


Meaning 


Post systolic 

Post continuous 

Murmur louder. than gr 3/6 (10 mm) 
Accentuated Py 

Diminished P2 

Fixed split Po 

Femoral pulse less than brachial 
Atrial fibrillation or broad 
notched P wave 

Axis, right (more than 110%) 
Axis, left (less than 0°) 

R wave greater than 1.2 mv in 
lead V, 

rR’ or qR in lead Vy 

R wave greater than 2.5 mv in 
lead V6 

T wave inversion in lead Vg 
Rib notching 

Peripheral vessels increased 
Peripheral vessels decreased 
Hilar vessels increased 

Hilar vessels decreased 

Main pulmonary artery large 
Main pulmonary artery not seen 
Aorta large 

Aorta small 

Cardiomegaly 

Snowman 


TEST50O 
TEST52 
TEST54 
TEST56 
TEST57 


cereal oe ok et 


TABLE 13 


Tests for Heart Disease Diagnosis 


Possible Results 


$02, S03 
$05, S06, S07, N 


n 
vi 
- 
J 
ZAmAzm 
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TABLE 14 


Diagnoses Based on All Available Attributes 
for Actual Heart Disease Case Histories 


4 


Posterior Distribution* Definitive Diagnosis 


DO9 


4 D20 0,41 NORMAL 


5 DO8 0.94 D33 


6 D32 (0.98 D32 


7 D31 0.47 D31 


8 D30 (0.87 D30 


9 D31 0.70 D27 


* Only diseases with probability greater than or equal to 0.01 
are shown, 
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Experiment 5. Sequential Diagnosis of Heart Disease Cases 

The actual heart disease cases were also diagnosed by the pro- 
gram using the sequential mode of operation. In each case, the 
initial attributes presented to the program were the results from 
a set of seven tests relating to physical signs. The diseases 
were assumed to be equally serious (1i; = 100,000, i # j) and all 
tests were assigned unit cost, The search depth in the test se- 
lection function was one in each case, 

A summary of the results of this experiment is presented in 
Table 15. Again, the advantage of sequential diagnosis is appar- 
ent, The program required an average of 5.8 tests to obtain a 
diagnosis compared to the thirty-three tests required to determine 
all attributes, This small number of tests is interesting. Re- 
call the sequential diagnosis of the bone tumor cases required an 
average of 6,7 tests per case, although the problem involves only 
one quarter as many states as the heart disease problem, Several 
reasons might be advanced to account for this, First, the tests 
associated with heart disease may include a number which have little 
value in differentiating groups of diseases, Thus, in a given 
problem, the test selection function may choose a terminal decision 
after relatively few tests have been run, A second reason may be 
the relevance of more inter-attribute relationships in the heart 
disease problem, Such relationships may be quite useful in diagno- 
sis 


, but the testing sequences for them are not examined since the 
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TABLE 15 


Sequential Diagnosis of Actual Heart Disease Cases 


Case and Number of Distribution Distribution 
Definitive Tests Selected at Terminal Based on all 
Diagnosis by Program Decision Attributes 
1. DO9 10 NORMAL 0,04 NORMAL 0.04 
DO2 0.06 DO3 «(0,91 
DO3 0.69 D34 0,03 
Dll 0,02 
D18 0,05 
D26 0.03 
D34 0.03 

2, DO4 4 DO2 0.08 DO2 0.09 
DO4 0.17 DO4 + 0.03 
DO5 0,62 DOS 0.83 
D31 0.10 D31 0.03 

3. DO2 1 D27 0,03 D32 «1,00 
D32 «0.96 

4, NORMAL 10 NORMAL 0,07 NORMAL 0.22 
DIO 0.03 D28 0.38 
Dll 0.07 D24 0,04 
D12 0.02 D20 «(0.41 
D20 0.67 D34 0,02 
D24 0,01 Dll =O.01 
D28 0.10 

5. D33 3 DOs 0,92 DOs 0,94 
D33. «0.01 D33. «0.05 

6, D32 0 D320. 98 D32. «0,98 
D29 ~0.01 D29 0,02 

7. D31 10 DO4 =0.01 D31 (0.47 
DOS 0.09 D30 «(0.37 
D31 0,86 DOS 0.08 
D32 —-0.02 D32 «0.02 

8, D30 8 DO2 0.03 D30 «(0.87 
DO5 0,02 DO2 ~+0,12 


9, D27 


6 Dil 
D19 
D24 
D26 
D31 
D33 


Average number of initial attributes 
Average number of tests by program = 


0,02 
0,01 
0.06 
0.06 
0.77 
0,03 


D31 0.70 
D27) 0.20 
D26 0,10 


= 7 
5.8 


135 


136 


depth of the tree search is limited to one level, Unfortunately, an 
increase in the depth of search leads to prohibitive amounts of com- 
putation in the heart disease problem. A deeper search may be possi- 
ble if more powerful breadth-limiting heuristics are developed, 

On the whole, the performance of the program with sequential 
diagnosis is comparable to that when all attributes are available. 
The one apparent exception to this involves case 9, Here the se- 
quential diagnosis failed to assign a probability of greater than 
0.01 to disease D27, The seriousness of this failure depends on 
medical considerations which are not discussed here, The general 
problem of measuring diagnostic performance, however, will be dis- 


cussed in Chapter 8, 


Chapter 7 


FURTHER EXPERIMENTS WITH THE DIAGNOSTIC SYSTEM 


In order to explore the potential value of the diagnostic 
system as a tool for the study of a variety of diagnostic problems 
and strategies, some further experiments were performed, The re- 


sults of these experiments are reported in this chapter. 


Experiment 6, The Effect of a Very Serious State 

In the experiments discussed in Chapters 5 and 6, it was 
assumed that the loss for misdiagnosis was the same for all pairs 
of diseases, For each experiment, the elements of the loss func- 
tion matrix were taken to be 0 for 1;; and 100,000 for lij 1#j. 
For this reason, the diagnostic program always selected the most 
likely disease as its terminal decision. One can easily imagine 
situations, however, in which the assumption of a constant loss for 
misdiagnosis independent of the actual disease is unrealistic, For 
example, it may be far more serious to diagnose pneumonia as a com- 
mon cold than vice versa, Since the diagnostic program incorporates 
such considerations in its rules for selecting a terminal decision, 
changes in the loss function matrix can result in pronounced 


changes in its decisions, 
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This effect was observed in two different situations, In 
the first, the loss function matrix is presented in Table 16. Note 
that it is very costly to miss the diagnosis of CB, The misdiag- 
nosis of either CS or ES as a disease other than one of these two 
or CB is quite serious, but it is not particularly serious to diag- 
nose CS as ES or CB or ES as CS or CB, Failure to diagnose one of 
the remaining diseases results in a loss which is independent of 
the diagnosis made, 

The generator was used to generate seven case histories of 
bone tumor cases. Each case was diagnosed by the diagnostic pro- 
gram in the light of the new loss function, The results of this 
experiment are summarized in Table 17. From this table, it can 
be seen that the new loss function affects only one decision, that 
of case 3, In this case, the diagnostic program selected CB as 
the terminal decision in spite of the fact that GC (the actual di- 
sease) was more than three times as probable, The loss for diag- 
nosing CB as GC is 1,000 times that of diagnosing GC as CB, however, 
and this fact dominates the decision of the program, The relative 
seriousness of CB does not affect the diagnoses of the remaining 
cases because the observed attributes excluded CB as a possibility 
in each case. 

The effect of a serious disease on diagnosis can be made even 
more pronounced if the serious disease is not easily distinguished 


from other less serious ones, For example, the disease CS often 


Diagnosis 
CB 
cs 
ES 
FS 
GC 
Os 
PS 
RC 


CF 


Loss Function Matrix for Bone Tumor Diagnosis 
(in thousands) 


TABLE 16 


Actual Disease 


FS 


1 


1 


GC 


1 


1 


os 


1 


1 


PS 


RC 


CF 
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TABLE 17 


Sequential Diagnosis of Cases for Loss 
Function of Table 16 


Number of Distribution 

Case and Number of Ini- Tests Selected at Terminal 
Disease tial Attributes by Program Decision 
1. (PS) 15 1 PS* 1.00 
2. (GC) 8 7 Gc* 0.90 
FS 0,09 
cs 0,01 
3, (GC) 9 3 CB¥ 0,24 
GC 0.76 
4. (ES) 10 0) ES* 0,99 
RC 0.0L 
5. (ES) 8 2 ES* 0,96 
cs 0.02 
6. (OS) 13 0 os* 1,00 
7. (GC) 8 12 GC* 0.89 
FS 0.09 
cS 0,02 


* Terminal decision by program. 


141 


appears in a terminal distribution when the actual disease is another, 
This means that CS has not been excluded as a possible diagnosis when 
a terminal decision is made, By making CS very serious relative to 
the other diseases, the decisions of the program can be strongly in- 
fluenced. 

The loss function matrix presented in Table 18 represents just 
this situation, A series of simulated cases was diagnosed by the pro- 
gram using this loss function. The results of this experiment are 
summarized in Table 19, Here the seriousness of CS dominates all 
decisions, and the terminal decision is CS in all cases. Note also 
that the terminal decision is made after relatively few tests have 
been run and while the posterior distribution is relatively diffuse. 
The predominance of terminal decisions for disease CS is a result 
of the seriousness of that disease. The decrease in the number of 
tests per case and the diffuse terminal distributions reflect the 
difficulty finding a single test which promises to significantly al- 
ter the expected loss, Since the diagnostic program employed a one 
level look ahead in searching the decision tree for these cases, it 
did not consider possible sequences of several tests to resolve this 
problem, This point will be discussed in more detail later in the 
thesis, 

The above example is but one in which the loss function has a 
significant effect on the terminal decisions made by the diagnostic 
program, Because the test selection strategy also accounts for the 


loss function, it, too, is affected by changes in the matrix, There- 
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Diagnosis 
CB 
cs 
ES 
FS 
GC 
os 
PS 
RC 


CF 


TABLE 18 


Loss Function Matrix for Bone Tumor Diagnosis 
(in thousands) 


100 


100 


cs 


ES 


Actual Disease 


FS 


1 


1 


GC 


1 


1 


1 


os 


1 


PS 


RC 


CF 
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fore, an important facility in the study of diagnostic strategies 

for a particular application is the ability to assess the sensitivity 
of these strategies to the loss function, Although the current 
version of the diagnostic system restricts the loss function to a 
matrix form, it is still possible to employ wide ranges of the 

values of the matrix elements in a given application study. This 
facility coupled with the capabilities of the generator makes it 
possible to study the performance of different versions of the diag- 


nostic program with a variety of matrix loss function. 


Experiment 8. Studies of a Test-Selection Heuristic 

The experiments discussed in Chapters 5 and 6 indicate the value 
of sequential diagnosis in reducing the number of tests required for 
a diagnosis, Therefore, it is worth some effort to improve the opera- 
tion of the test-selection function, 

One problem which can arise in the use of the test-selection 
function of the current system is the appreciable amounts of com- 
putation required to evaluate all the relevant tests at a given de- 
cision node, It would be quite desirable to reduce the amount of 
computation devoted to test selection provided that the diagnostic 
capability of the program were not impaired, As an example of the 
amount of computation involved in test selection, consider the fol- 
lowing. In the diagnosis of congenital heart disease, there can be 
as many as thirty-five states with non-zero probabilities in the 


current distribution, If there are twenty relevant tests at a given 
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decision node, each with two possible results, a one-level evaluation 
of these tests could require the creation of forty distributions, 
each requiring the computation of thirty-five updated probabilities, 
This is a significant amount of processing for a highly interactive 
program, and the example cited does not represent a particularly 
large set of alternatives, Since the test-selection function may be 
performed many times during a diagnosis, there is a good reason to 
reduce the time required to perform it. An obvious approach is to 
improve the efficiency of the code for the function. While this 
would no doubt lead to improvements, it was not attempted. Atten- 
tion was focused on attempting to reduce the number of tests con- 
sidered, rather than reducing the time devoted to the evaluation 

of an individual test. 

This approach was motivated by the results of the experiments 
with sequential diagnosis, There it was observed that relatively 
few tests were required for diagnosis by the program. The particu- 
lar set of tests employed for a given diagnosis is determined dynam- 
ically by the program, and varies from one diagnosis to another, If 
one could guess which tests would be relevant to a particular diag- 
nosis, the total number of tests considered could be reduced signifi- 
cantly. A guess about the relevance of certain tests must not be 
irreversible, however, because the value of some tests will become 
apparent only after other tests have been run, 


At any stage in a diagnosis, the current distribution provides 
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the most logical basis for a hypothesis about the future relevance 
of particular tests. One heuristic which incorporates this view is 
the one which restricts the set of tests considered to those which 
are relevant to the state which is the best terminal decision at the 
current node, This heuristic favors those tests which tend to con- 
firm or disprove the current “best guess" about the problem, It 
also had the property of reversibility mentioned above, When the 
terminal decision changes, the set of relevant tests changes corres- 
pondingly. 

This heuristic was employed in a number of experiments with 
both congenital heart disease problems and bone tumor problems. In 
the cases studied it resulted in the same number of tests selected 
as the standard function which employs no such heuristic, This 
heuristic does reduce the average number of decision nodes considered 
per diagnosis, This reduction is not great, however, because in both 
problem areas the diseases share many attributes in common, and hence 
many relevant tests, Thus, at any decision node, almost almost all 
the tests are relevant to the state determined to be the best terminal 
decision, 

A second heuristic which offered a potentially greater reduction 
in the number of decision nodes considered per diagnosis was also 
considered. This heuristic employs the current distribution to "guess" 
which tests will not be useful in the remainder of the diagnosis, 


Tests which are thought to have little value are temporarily removed 
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from consideration, At a later point in the diagnosis these tests 
may be released for further consideration, 

The actual operation of this heuristic is as follows, At a 
given decision node, the set of relevant tests is evaluated by the 
test selection function, Then the set of ‘tests is partitioned into 
two disjoint subsets, Im the first are all those tests with the 
property that the sum of the cost of the test plus the expected loss 
of a terminal decision after the test has been run exceeds the ex- 
pected loss of the current terminal decision, These tests are said 
to be dominated, The second set consists of all the remaining un- 
dominated tests, The heuristic hypothesizes that. the tests in the 
dominated set wilI remain dominated for the remainder of the diag- 
nosis, This set of tests is placed on the top of a push-down stack. 
At each decision node the push-down stack is examined prior to evalu- 
ating each test. If the test is found in the stack it is not con- 
sidered at the decision node, 

In general, then, each iteration of the test selection function 
produces a new set of dominated tests which are pushed onto the stack, 
This means the set of relevant tests is generally decreased at each 
stage of the diagnosis, Whenever there are no undominated tests at a 
given decision node (i.e, whenever the terminal decision is selected), 
the program releases the set of dominated tests (1f one exists) on 
the bottom of the stack, This corresponds to re-evaluating those 


tests which were tentatively discarded earliest in the diagnosis, 
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The reason for this choice is that it is desirable to reconsider 
tests which were dominated when the distribution was quite different 
from the present one, If the distribution has changed little, tests 
which were formerly dominated are apt to be currently dominated, Ac- 
tually, there is no guarantee that this method will produce the de- 
sired effect, It is used primarily as an example of a possible ap- 
proach, and additional discussion will be devoted to the subject below, 

The "dominated-test" heuristic was tested in the sequential 
diagnosis of both the heart cases and bone tumor cases, The nine 
heart disease cases and the twelve bone tumor cases were used as the 
testing sample, The same initial attributes for a given case were 
given to both the "'dominated-test" heuristic and the standard ver- 
sion of the diagnostic program, The number of tests by the program, 
the number of decision nodes considered during diagnosis, and the 
distribution at the terminal decision were all recorded, These re- 
sults are summarized in Tables 20 through 23, A number of these 
results have an interesting interpretation, 

In both the heart disease cases and the bone tumor cases, the 
dominated-test heuristic results in a substantial reduction in the 
average number of decision nodes considered per diagnosis. In the 
heart disease problem, this heuristic results in a larger average 
number of tests performed per diagnosis, In situations when the cost 
of an average test exceeds the value of the computation saved, this 
is an undesirable effect. The reason for this reduction in diag- 


nostic efficiency can be seen from the following interpretation of 
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TABLE 20 


Sequential Diagnosis of Heart Disease Cases-- 
Standard Test Selection Function 


Number of 
Number _ of Decision Distribution 
Case and Initial Tests Selected Nodes at Terminal 
Diagnosis Attributes by Program Considered Decision 


1, DOS 7 10 541 NORMAL 0.04 


2, DO4 7 4 287 DO2 


3, DO2 7 1 133 D27 


4, NORMAL 7 10 523 NORMAL 


5. D33 7 3 248 DO8 
6, D32 7 0 66 D32 (0.98 


7. D31 7 10 513 B04 0.01 
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8. D30 


9, D27 


7 8 457 DO2 
DOS 
D20 
D30 


7 6 379 Dil 
D119 
D24 
D26 
D31 
D33 


Average number of tests by program = 5.8 
Average number of decision nodes considered = 350 


TABLE 21 


Sequential Diagnosis of Heart Disease Cases 
Dominated Test Heuristic 


Number 
of Tests Number of 
Case and initial Selected Decision Nodes 
Diagnosis Attributes by Program Considered 
1, DOO 7 11 283 
2. DO4 7 5 163 
3. DO2 7 1 66 
4. NORMAL 7 16 345 
5. D33 7 3 176 
6, D32 7 0 66 
7. D31 7 11 269 


Distribution 
‘ at Terminal 


. Decision 
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8, D30 7 10 301 DO2 
DO4 
DOS5 
D18 
D20 
D30 
D31 
D32 


9, D27 7 6 216 D11 
D31 
D24 
D26 
D33 


Average number of tests by program = 7 
Average number of decision nodes considered = 208 


0.04 
0,02 
0,02 
0.03 
0.04 
0.70 
0,08 
0,02 


0.02 
0.77 
0,06 
0.06 
0,03 
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TABLE 22 


Sequential Diagnosis of Bone Tumor Cases 
Standard Test Selection Function 


Number _ of Number of 
Case and Initial Tests Decision Distribution 
Pathology Attributes Selected by Nodes at Terminal 
Program Considered Decision 
1. (GC) 7 9 269 GC 0.78 
cB OO,21 
2. (OS) 10 12 425 os 0.35 
cs 0,65 
3. (CB) 9 0 0 CB 1.00 
4, (CS) 70 4 223 cs 0.99 
5. (0S) 10 4 194 os 1.00 
6, (RC) 10 13 406 RC 0.68 
ES 0.30 
FS 0.01 
7. (CS) 8 4 228 cs 0.78 
FS) 0,22 
8. (ES) 8 11 475 ES 0.05 
FS 0.07 
RC 0.87 
9. (ES) 6 5 278 ES 0.88 
RC 0,05 
os 0.05 
cs 0,02 
10, (CB) 10 3 109 CB 0.96 
CF 0.04 
ll. (GC) 10 5 169 Gc 0.81 
cs 0.10 
CF 0.08 
ES 0,01 
12, (PS) 10 3 142 PS 0.93 
FS 0,02 
cs 0.05 


Average number of tests by program = 7.1 
Average number of decision nodes considered = 243 


[er ner te 
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TABLE 23 


Sequential Diagnosis of Bone Tumor Cases 
Dominated Test Heuristic 


Number 
of tests Number of Distribution 

Case and Initial Selected Decision Nodes at Terminal 
Diagnosis Attributes by Program Considered Decision 
1. (GC 7 7 151 CB 0.73 
Gc 0.26 

2. (OS) 10 17 211 cS 0.66 
os 0.34 

3. (CB) 9 0 0 cB 1,00 
4. (cS) 10 5 148 cs 0.82 
ES 0.09 

Fs 0.05 

os 0.05 

5. (OS). 10 3 139 Os 0.92 
ES 0,02 

cs 0.03 

RC 0.03 

6. (RC) 10 14 218 RC 0,70 
ES 0,29 

7. (CS) 8 4 180 cS 0.74 
FS 0.26 

8. (ES) 8 15 294 RC 0,90 
FS 0,05 

ES 0,03 

9. (ES) 6 5 137 ES 0.87 
cs 0,03 

FS 0.01 

OS 0.04 

RC 0,04 

10. (CB) 10 3 97 CB 0.96 


ll, 


12, 


(GC) 


(PS) 


10 5 119 GC 
cs 
ES 
CF 


10 3 106 PS 
cs 
FS 


Average number of tests by program = 6.6 
Average number of decision nodes considered = 150 


0,81 
0,10 
0.01 
0,08 


0,92 
0,05 
0.02 


155 


156 


the heuristic, 

This heuristic simulates to a certain extent the diagnostic 
strategy of one who seizes upon an initial view of the problem and 
later yields that view with considerable reluctance, Thus, the 
program makes a guess as to which tests will prove important at an 
early stage in the diagnosis, and thereafter restricts its attention 
to those tests as long as some appear to be useful, The difficulty 
is that the view on which the guess was made may not be an accurate 
one, Although the tests being considered may be of some value, 
there may be other tests, temporarily disregarded, which may be of 
greater value, Unfortunately, the heuristic is not sufficiently 
sensitive to changes in the current distribution, and it may cause 
relatively unfruitful paths to be pursued to an unnecessary extent. 
When it eventually abandons such a path and re-evaluates the formerly 
dominated tests, it may already have incurred unnecessary testing 
costs, The heuristic exhibits a "single-mindedness" which results 
in less than satisfactory performance, 

In the bone tumor cases, this heuristic reduced both the 
average number of decision nodes considered and the average number 
of tests run, Here its failing is a loss of accuracy, This effect 
is extremely interesting. Apparently in its pursuit of an informa- 
tive series of tests, the program succeeds in obscuring much of the 
information implicit in the initial attributes. As a result, when 


the undominated tests are finally released for consideration, the 
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current distribution is sufficiently altered that the program does 
not find additional tests worthwhile. This effect may be the cause 
of the results for case 1, Here the dominated test heuristic se- 
lected fewer tests in arriving at a less satisfactory diagnosis than 
the standard test selection function. 

While the heuristic in question has some shortcomings, it does 
indicate a certain amount of promise, What it seems to lack is an 
awareness of changes in the current distribution which should cause 
certain dominated tests to be released for consideration, One possi- 
ble solution is to save the current distribution with a set of 
dominated tests, This would allow the program to compare the pres- 
ent distribution with one in the stack to determine whether the view 
of the problem has changed sufficiently to warrant the release of 
the tests, This comparison could also account for the relative 
seriousness of states in deciding whether a given change were sig- 
nificant, 

This example is but one of a number of heuristics which can be 
studied in the diagnostic system, Because very large decision trees 
may be encountered in future applications, a variety of tree-pruning 


heuristics should be studied, 


Experiment 9, Exercise of the Pattern-Sorting Capability 
A small example was constructed with which the pattern-sorting 
capability could be tested, This example consisted of six states 


and fifteen attributes. The matrix for the example is presented in 
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Table 24, The states in this example can be partitioned into two 
sets which have the property that certain attributes are specific 
to the states in a group and other attributes are shared by the two 
groups, The generator was employed to simulate case histories with 
noise attributes, That is, a case history for a state in the first 
group included one or more attributes selected from those specific 
to the states in the second group, 

Consider the following diagnostic problem with the loss func- 
tion as specified in Table 25, The initial attributes are $10, S12, 
S13 and SO4, These attributes cannot be attributed to a single state, 
and so the pattern-sorting function produces more than one pattern. 
In this case the patterns formed are (S04) and (S10, $12, S13). For 
each of these patterns the distribution over states is obtained as- 


suming that the given pattern is the only one, These distributions 


are: 

1) (S04): DONE 0.24 2) (810,812,813): DFOUR 0.42 
DTWO 0.11 DFIVE 0,02 
DTHREE 0.65 DSIX 0.57 


Based on these distributions, the pattern-sorting function selects 

the current pattern. Here the choice is pattern 1 although it con- 
tains only one attribute, From the loss function matrix, it can be 
seen that state DONE is very serious, Since state DONE can exhibit 
S04, the posterior probability of DONE given SO4 is non-zero (0,24), 


By considering both posterior probabilities and losses, the pattern- 


A priori 
Probability 


Disease 


DONE 


DIWO 


DTHREE 


DFOUR 


DFIVE 


DSIX 


sol 


.10 
Pate: 
.60 
.10 
<25 


.40 


S02 


.50 


35 


20 


.80 


35 


235 


S03 


~40 
250 
»20 
. 10 
-40 


229 


TABLE 24 


Artificial Structure 


S04 


.30 
.10 


.80 


S05 


.70 


.20 


.05 


S06 


. 10 
70 


,00 


S07 


.20 
.05 


.05 


S08 


.10 
.90 


.50 


S09 


.05 
.10 


.60 


$10 


.30 
25 


235 


$11 


. 10 
a29 


.90 


$12 


725 
705 


.25 


$13 


.30 
.10 


Peo) 


.05 


.60 


,29 


$15 


$14 


.20 
.00 


.60 
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TABLE 25 


Loss Function Matrix for Six State Problem 
(in thousands) 


DONE DIWO DTHREE DFOUR DFIVE DSIX 
DONE 0 1 1 1 1 at 
DTWO 100 0 1 1 1 ] 
DTHREE 100 l 0 1 1 
DFOUR 100 L a 0 1 
DFIVE 100 1 vf 1 0 


DSIX 100 1 1 L 1 0) 
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sorting function selects pattern 1 as the more serious, and hence 

it becomes the current pattern, Tests are selected relative to this 
pattern, but any new attributes are processed through the entire pat- 
tern stack as discussed in Chapter 4. In this particular example, the 


program continued diagnosis until the following situation was obtained: 


ls (S02, S04, NOT S06, SO7, NOT SO8, NOT S09) 


DONE 0,92 
DTHREE 0.08 


2. (S10, $12, $13, S02) 


DFOUR 0.62 
DFIVE 0.01 
DSIX 0.37 


The program then tentatively attributed pattern 1 to state DONE, 
This left S10, $12, and S13 unaccounted for, At this point, the 
user terminated the diagnosis, Had he wished, he could have pursued 
the investigation, the original pattern was shown to be invalid, the 
attributes in it would be returned to the unaccounted-for set and 
the pattern would be removed from the stack, 

A variety of such experiments were run with the pattern-sorting 
function and the results indicated that the particular scheme embodied 
in the function exhibits the desired properties, This function needs 
to be studied more extensively, however, especially in more complicated 
situations, Although this area was somewhat slighted in this research 
the environment provided by the diagnostic system should be a good 


one in which to pursue such a study. 


Chapter 8 


DISCUSSION OF THE RESEARCH 


The research discussed in the preceding chapters suggests 
a number of questions and issues which merit additional comment, 
In this chapter an attempt is made to draw together a number of 
results and to consider their potential generality, Also of interest 
here are some of the possible extensions of this research which aim 
at developing a more sophisticated system for the study and perform- 
ance of diagnosis. 

One of the more obvious questions involves the evaluation of 
the performance of the current diagnostic program. This question 
is important for two reasons, First, one of the principal hypothe- 
ses considered in this research was that in a variety of problem 
areas, a computer program could prove a competent or superior diag- 
nostician, The current program has been applied to a number of 
cases, simulated and actual, of bone tumor and congenital heart di- 
sease, Hence a reasonable question is how well did it perform. A 
second reason for establishing a meaningful performance measure is 
so that it can be used in studies of various diagnostic strategies, 
If one test selection heuristic is to be judged superior to another, 
the judgment must be based on a measure of performance, and that 


measure should reflect diagnostic capability, So there is a very 
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real need for a good measure of diagnostic performance. 
Unfortunately, while the need for a performance measure is clear, 
the precise nature of such a measure is open to a number of ques- 
tions, Perhaps the best way to approach the problem is to catalog 
those qualities for which a diagnosis is generally judged to be a 
good one, The most obvious of these qualities is the accuracy of the 
diagnosis, The object of diagnosis as stated in the beginning of 
this thesis is to ascertain the state of a system, All other things 
being equal, the more accurate the determination of the state of the 
system, the better the diagnosis, By itself, however, this quality 
has relatively little meaning. One desires to know the state of a 
system in a diagnostic problem because this knowledge is an input to 
a subsequent decision (e.g. the decision about a treatment plan for 
a medical problem), Accuracy is not sought for its own sake, but 
rather for its improvement of decisions which result from the diagno- 
sis, If these latter decisions are independent of any particular al- 
ternative in a group of diagnostic decisions, then there is no bene- 
fit to be accrued from distinguishing one of this group from another, 
From the point of view of further decisions, the states corresponding 
to these decision alternatives constitute an equivalence class, If 
a doctor knows that a patient has one of three viruses, all of which 
would be treated in the same manner, there may be no value attempt- 
ing to deduce the "actual" virus, 


If one were interested in accuracy as the chief quality of good 
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diagnosis, he could contend that in the above example, the doctor 
was accurate in diagnosing the problem as one of three viruses and 
that this can be thought of in identifying the state of the patient, 
A simple extension of this example makes this objection less foceeear 
however, Suppose that each of the three viruses are treated in a 
different manner and that there is a loss of diagnosing any one as 
another, but in each case this loss is less than the testing loss 
required to distinguish one from another, Again the identification 
of the goal of diagnosis as accuracy seems incomplete, The point is 
that accuracy is sought only to an extent commensurate with the ex- 
pected consequences of a diagnostic decision about the system and 
the expected cost of obtaining greater accuracy, 

This view of the diagnostic process has been the basis for this 
research, From the point of view of the diagnostician, the goal 
of diagnosis is to minimize the sum of the testing loss and the ex- 
pected decision loss. Conceivably a diagnostician could correctly 
ascertain the state of a system at such a testing cost that his diag- 
nosis would be judged inferior, 

While it is appropriate for a diagnostician to consider expected 
loss for misdiagnosis as a factor in determining the course of a 
diagnosis this quantity is not necessarily relevant to the judgment 
of his diagnostic performance, The principal reason for this is that 
the expected loss depends on the probability distribution over states 


which is held by the diagnostician at the time of a terminal decision, 
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Since the diagnostician chooses tests, this distribution reflects 
his testing strategy as well as the actual problem, Basing a per- 
formance measure on expected loss ignores the relative merits of dif- 
ferent testing strategies, It is as though a doctor were to be given 
a high performance rating simply because he believed very strongly 
that he had discovered the patient's problem, This strong belief may 
well be founded on incomplete or irrelevant information, 

A more satisfactory way of assessing diagnostic performance is 
to simply add the testing loss to the actual decision loss, That 
is, judge the act rather than the intent. Ideally, one could deter- 
mine the actual decision loss by comparing the actual state of the 
system (when it becomes known) with the diagnostic decision and 
determining the loss attributable solely to the difference between 
the two, By this standard, a diagnostician who consistently mini- 
mized the sum of testing and decision losses would be judged to be 

superior. Some of the problems inherent in this measure are rather 

obvious, First, the actual state of the system may never be known 
with certainty. A patient who is diagnosed and treated may never 
return for further examination, and hence a serious misdiagnosis may 
never be uncovered, A second problem is the difficulty in appor- 
tioning the decision loss to various diagnostic decisions, Also, 
the loss itself may be very difficult to ascertain, Nonetheless, 
this measure does seem to subsume the desired properties, and al- 
though it may be difficult to apply, it does seem to be a standard 


to be sought, 
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Another consideration in evaluating diagnostic decisions couched 
in terms of probabilities is the interpretation of probability dis- 
tributions. For example, what are the implications of a diagnosis 
of (0.75, 0.25) for the states S] and S2 for a performance measure? 
To a large extent, it depends on the actions which are taken based 
on this diagnosis, Suppose the actual state is $2. How does this 
affect the evaluation of this diagnosis? If only a single action 
can be taken on this diagnosis and it is based on the belief that the 
state is Sl, the problem is even more difficult, The influence of 
such a distribution on a human decision maker may be quite subtle. 

If individuals react differently to such distributions, the problems 
will be compounded. 

Finally, some effort should be made to normalize performance 
measures, Certain problems may be inherently more difficult to 
diagnose than others, For this reason, it is important to obtain 
an understanding of the limitations placed upon even the most expert 
diagnostician by the very nature of the problem before him, 

The evaluation of the performance of the diagnostic program 
in the particular problem areas of bone tumors and congenital heart 
disease is made more difficult by the lack of well-defined loss 
structure for these problems. This precludes the use of the total 
loss measure discussed above. An alternative approach is to compare 
the program performance with standards based on the performance of ex- 
perienced doctors, Even this approach is somewhat indirect in this 


case, Since no studies of doctor performance with the particular 
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case histories used were performed, no immediate comparisons based 
solely on the results of this research are possible. Some indica- 
tion of program performance, however, can be obtained in the follow- 
ing way, The problems of bone tumor diagnosis and heart disease 
diagnosis have been studied extensively by Lodwick and Warner res- 
pectively, Both developed computer programs to perform diagnosis 

and have compared the performance of these programs with that of ex- 
perienced physicians, These comparisons suggested that the programs 
performed diagnosis of a quality comparable to that of an experienced 
physician when all attributes were presented to both physician and 
program, The fact that the current diagnostic program duplicates 

the results of these programs on the cases studied suggests that the 
current program would fare equally well in a comparison with physicians, 
In the absence of a performance measure, this is the strongest state- 
ment which the experimental evidence will support, 

If one tentatively accepts this suggestion, then a second sig- 
nificant conclusion can be derived from the results of these experi- 
ments, The diagnostic program was able to solve problems in two 
different areas of medical diagnosis. These areas differ in both 
the number of diseases and the complexity of inter-attribute rela- 
tionships which are considered, The latter aspect is particularly 
important because it was handled without changing the program. Since 
the experiments involved only two problem areas and both were medical, 


the applicability of the program for a wide class of problems has 
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not been established. Its success in the two areas mentioned, 
however, strengthens the belief that it does have wider applica- 
bility. 

The fact that the program is independent of the content of 
the information structure might be of significant value in the use 
of the program with hierarchical structures, Consider, for example, 
the problem of diagnosing a very large set of diseases, One possi- 
bility would be to create a hierarchical structure in which many 
sub-structures exist. The structures for bone tumors and congenital 
heart disease might such sub-structures, At the higher levels, 
the states would be classes of diseases, such as heart disease, 

The goal of diagnosis at higher levels would be to determine the 
proper class of disease. When this determination had been made, a 
more detailed sub-structure for that disease class would be employed 
for a "finer" diagnosis. The same diagnostic program could deal 
with all sub-structures, This would be a great improvement over a 
large set of programs, one for each sub-structure. 

Again, considering the results of diagnosing actual case his- 
tories, one can readily appreciate the advantage of sequential diag- 
nosis, In the particular problems studied, the program was able to 
arrive at a diagnosis with the use of relatively few tests, This 
capability is very important since the testing cost for a diagnosis 
may be a significant part of the total cost. Tests which are un- 


necessary or uninformative may exact a high price, and an effort 
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should be made to restrict the tests run to those essential to the 
diagnosis, The sequential test selection facility permits the pro- 
gram to dynamically assess the potential usefulness of each possible 
test. This results in efficient testing strategies, an important 
component of good diagnosis. 

In a problem area in which the tests relevant to different 
groups of states are relatively disjoint, the value of sequential 
testing should be even greater. Once the appropriate group of states 
has been established, the tests considered can be restricted to the 
set of tests associated with that group. In the absence of a sequen- 
tial testing capability, it may be necessary és perform all tests to 
obtain information which could have been obtained from a few, The 
striking reduction in the number of tests required for diagnosis of 
bone tumors and congenital heart disease effected by sequential testing 
strongly suggests the potential value of this approach in other diag- 
nostic problems, 

The existence of a diagnostic system rather than just a diagnos- 
tic program has proved quite important in this research, Many of 
the strategies which were considered are quite complicated, and it is 
difficult to predict a priori the manner in which they will perform, 
The generator has been very useful in testing these strategies under 
a variety of problem conditions, Also of use has been the facility 
for selectively monitoring particular diagnostic functions such as 


pattern-sorting and test selection by collecting detailed data on 
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their operations, 

One virtue of the inclusion of a generator in the diagnostic 
system is that it makes it possible to study the performance of the 
diagnostic program in problems derived from a wide range of informa- 
tion structures, The simulation capability frees the researcher 
from dependence on actual case histories, Thus he can create struc- 
tures and simulated cases specifically designed to test some aspect 
of the diagnostic program, The use of the simulation facility with 
an information structure corresponding to an actual diagnostic 
problem may also be very useful in the study of that particular 
problem. 

Complementing this capability is that of operating the diagnos- 
tic program in an interactive mode, Thus a user can employ the pro- 
gram in actual diagnostic problems, This “open end" of the system 
permits the independent testing of strategies developed through re- 
search, as well as making the diagnostic program a practical aid to 
problem solving. The experience gained in this research indicated 
the value of such a system which permits the study of both actual 
and artificial diagnostic problems, It seems that this type of 
system would prove must useful in further development of sophisti- 
cated strategies for computer-aided diagnosis, 

Finally, the modularity of the system is very important, On 
the one hand, the insulation of the system functions from one another 
permits one to study a wide variety of diagnostic strategies since 


the functions can be changed independently of one another, Also as 
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better versions of these functions are developed, they can be incor- 
porated into the system without restructuring it. In this sense, the 
performance of the system can be improved as additional experience 
with it is obtained, 

The experience obtained with the diagnostic system has pointed 
to a number of areas for further research, A number of these areas 
are discussed here, Some pertain to specific improvements in the 
diagnostic capabilities of the program, while others have more gen- 
eral ramifications. 

In Chapter 7, certain experiments to study the effect of the loss 
function on diagnosis were discussed, While these experiments are by 
no means exhaustive, they do indicate the strong effect the loss 
function can exert on diagnoses obtained by the program, Two major 
questions need to be investigated in this regard. The first is how 
such a loss function can be developed for a particular problem area,and 
the second is in what ways is diagnosis sensitive to the actual values 
of a loss function, 

The first question is a very difficult one to answer, Assuming 
for the moment that the matrix form of the loss function is retained, 
the problem is to determine the "seriousness" of each possible mis- 
diagnosis in some appropriate units, For example, in the context of 
medical diagnosis, one must answer questions such as "How serious is 
the diagnosis of pneumonia as influenza and vice versa?" This answer 


must be in such terms as to permit the comparison of a wide variety 
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of misdiagnoses in an orderly manner. If one considers the extreme 
range of consequences resulting from misdiagnoses in medicine, he 
can appreciate the magnitude of this task. As stated, the problem 
required the establishment of a common scale for such extremes as 
the failure to diagnose a simple cold and the failure to diagnose 
cancer, 

In many instances, the loss for a misdiagnosis depends on many 
extraneous factors, such as whether a patient will return to the doc- 
tor when his symptoms persist. The loss may also depend on decisions 
made after the diagnosis which are difficult to predict, Compound- 
ing the problem of the loss function is the need to convert the test- 
ing loss to the same scale, In particular areas, one may be confronted 
with further complications in this regard, For example, the question 
of a loss function for medical diagnosis is also a question of whose 
loss function should be employed, One could answer that the loss 
function should be that of the patient, The loss function of the doc- 


tor, and that of society, however, are also possible answers to this 


, 

question, If a diagnostic system were created for general use in 

medical diagnosis, questions such as these would have to be considered, 
Although the problems of determining the loss function for 

an area as complex as medical diagnosis would be very great, they 

may well prove worth the effort of solution, If the value of a pro- 


gram for diagnosis in a given area can be clearly demonstrated to 


be considerable, this would be strong motivation for work on an ap- 
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propriate loss function, As currently conceived, such a diagnostic 
program would make extensive use of losses in directing a diagnosis. 
These losses should reflect the best understanding of the conse- 
quences of possible decisions, In some areas, the development of a 
loss function might be a valuable exercise independent of the im- 
plementation of a diagnostic program. In areas where sophisticated 
diagnosis is currently being performed by human beings, a loss func- 
tion is often implicit. The attempt to quantify this loss function 
may reveal inconsistencies and reveal implicit losses of questionable 
merit, To the extent that this situation obtains in a particular 
area, there is additional motivation for research into this problem, 

Such research would involve investigation of means of quantify- 
ing and scaling diverse consequences as well as considerations of the 
best form which the loss function should take, To a large extent, a 
framework for these investigations has already been established. A 
number of workers in the areas of statistical decision theory, game 
theory, and economics (R21, R22) have considered many of the prob- 
lems associated with the attempt to scale decision alternatives, 
While this work is far from complete, it does provide a reasonable 
basis for some of the initial studies, This whole area is rich with 
problems of interest and importance, 

Another important area for research is the development of a 
diagnostic program which includes improved solutions to a number of 


different problems, some of which are discussed here, 
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As previously noted, the test selection function merits particu- 
lar attention. This function serves a central purpose in the over- 
all diagnostic strategy of the program, and as a result, significant 
improvements in this area would directly promote the diagnostic capa- 
bility of the program. More sophisticated test selection heuristics 
are required if the program is to deal successfully with problems 
involving large numbers of decision and testing alternatives, All 
the test selection heuristics employed in this research is "fixed- 
depth" in the sense that they explore all branches away from a given 
decision node to a fixed depth in the decision tree, Most likely a 
better test selection function would explore branches to varying 
depth, pursuing further those branches which appeared more promis- 
ing. The difficulty yet to be overcome in this regard is the es- 
tablishment of some measure of promise" for branches in the decision 
tree, This problem has been encountered in other applications of 
heuristic programming, and it can be expected that significant re- 
sults in the diagnostic problem would be of more general applica- 
bility. Similarly, if powerful test selection heuristics can be 
developed, they might be of considerable value in a variety of 
sequential decision problems, 

Another improvement to the diagnostic program would allow it 
to take advantage of various relationships among tests, For example, 
if one is going to perform a certain test, it may be advantageous 


to perform another test as well because it is inexpensive when run 
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in conjunction with the first test. The inclusion of more complete 
information about tests in the information structure might allow 

the program to exploit various inter-test relationships and to select 
groups of tests to be run during diagnosis, 

The pattern-sorting function needs to be bolstered by the addi- 
tion of facilities for assessing the accuracy of the attributes 
provided it by the user, Just as it is important to detect noise 
attributes, it is equally important that the presence of false in- 
formation be discovered, Undoubtedly only partial solutions to this 
problem are possible, but additional capabilities of this kind, 
even if somewhat limited, would be of considerable value in appli- 
cations of the program to actual diagnostic problems, For example, 
the program could include a means for incorporating estimates of 
the reliability of tests into both the pattern-sorting and inference 
functions, 

A number of improvements can be made in the inference function 
of the program, One of these is the incorporation of a learning 
scheme within this function, Such a scheme would permit the pro- 
gram to learn the a priori probabilities for the various states as 
well as the conditional probabilities of attributes of given states. 
Bayesian framework provides a convenient structure within which a 
learning scheme can be developed, Learning of this type is especially 
important if the relevant probabilities vary with the specific appli- 


cation, For example, if the information structure for congenital 
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heart disease were employed in a region of the country other than 
that in which it was developed, the probabilities might require ad- 
justment to reflect changes in the characteristics of the popula- 
tion of potential patients, The program can obtain the information 
required for such an adjustment from the actual diagnoses which it 
performs on patients from the new population provided that other 
means of obtaining diagnoses are available. Thus in certain appli- 
cations, the diagnostic program may require a training period in 
which it can alter the contents of the information structure to more 
accurately reflect the relevant behavior of the given system, A 
variety of learning schemes should be investigated to develop a 
scheme which will be suited for this problem, 

Some of the considerations involved in research of this kind 
are apparent at the outset. If the probabilities of interest are 
relatively stable, then a rather prolonged learning period may be 
acceptable in the hope that these probabilities will be learned accu- 
rately. On the other hand, if the probability structure of the 
problem is relatively dynamic, then more rapid learning may be re- 
quired, One difficulty with the latter situation is that rapid 
learning implies a greater weighting of recent experiences and if the 
environment is noisy, this may lead to poor probability estimates, 
and hence to poor diagnosis, One possibility is to exploit the 
ability of the human diagnostician to perceive patterns and trends 


by allowing him to influence probability estimates dynamically, For 
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instance, a doctor might be better able to detect the early stages 
of an epidemic and hence adjust the a priori probability of the 


prevalent disease to reflect its increased incidence. 


Some Comments on the Diagnostic Model 


When one devotes considerable attention to the problem of diag- 
nosis, he may experience a tendency to generalize his definition of 
the problem so as to encompass an increasingly wide circle of prob- 
lems, The danger of this tendency is that it may result in the ex- 
tensive discussion of diagnostic programs and systems of impressive 
capabilities which are founded more on wishful thinking than on 
practical experience, Because the appeal of such an intellectual ex- 
ercise is strong, it is important to consider carefully the model of the 
diagnostic problem being employed in order to obtain a realistic view 
of both its potential and limitations. Some of the important charac- 
teristics of the model employed in this research are investigated 
here with this intention, 

A diagnostic model based on attribute-state relationships has 
understandable appeal. In many diagnostic problems the most visible 
aspect of an expert's attack on a problem is his gathering of attrib- 
utes on which to base his decision. In many instances he may appear 
to relate these attributes directly to the possible states of the 
system. When the difficulty of diagnostic problems in general is 
considered, however, it seems unlikely that the human expert per- 


forms only a simple association of attributes and states to arrive 
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at a diagnosis, Diagnosis, as performed by humans, seems to be a 
subtle and often complex process of association and deduction, 

The model employed in this research, on the other hand, is 
very explicit in the way in which it relates attributes and states, 
Associations in the information structure are relatively direct, 
and deduction is performed in a uniform manner for all problems. 

In one sense, the model employed by the diagnostic program appears 
quite rigid and simple, Even this brief comparison with human diag- 
nosis suggests an important question, Can this relatively simple 
model be sufficient for a diagnostic program to perform effectively? 
A derivative of this question is the following. To what extent can 
a program based on this model be successful in performing diagnosis 
in a variety of problem areas? Although the evidence gathered from 
this research is far from sufficient to allow definitive answers to 
these questions, it does permit some insights into the problems to 
which these questions are addressed, 

The author believes that the basic functions developed in this 
work reflect aspects of a diagnostic program which has both potential 
generality and power, At present, the functions are quite crude in 
their structure and capabilities, but the conception of diagnosis in 
terms of these functions (or their more sophisticated successors) is 
believed to be both a useful and viable one. One problem may be that 
the current separation of functions is somewhat restrictive, but this 
has the advantage of emphasizing the principal objectives and problems 


of each, This emphasis is very important in the initial phases of 
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research in this area, and the separation permits the study of differ- 
ent versions of one function more or less independently of the others. 

In broad outline, the model incorporates the principal features 
of diagnosis as performed by human beings, The inference function 
coupled with the information structure allows the consideration of 
both past experience and current information in a particular diagnosis, 
Bayesian inference provides an orderly way for balancing these two 
elements in the deductive process, The test selection function pro- 
vides the program with a rational means for choosing tests which 
accounts both for their cost and their potential value in furthering 
the diagnosis, Finally, the pattern-sorting function provides a 
means for performing diagnosis in the presence of noise, 

While it is unlikely that the human diagnostician employs this 
particular division of the diagnostic function, the total capability 
incorporated in the functions seems to approximate that required, 

It is also important to note that there is no particular reason to 
require a diagnostic program to simulate the processes employed by 
humans, A more appropriate requirement is that a diagnostic pro- 
gram should allow the exploitation of the comparative advantages of 
a computer in order that the total diagnostic capability of a man- 
machine partnership may exceed that attainable by either above. 

For example, it has been noted that doctors do not organize 
their diagnostic experience into large lists of symptoms and diseases, 


but rather associate their experience with and through their under- 
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standing of the human body and its processes, It would be extreme 

to conclude from this that such an organization is a necessary one 
for diagnosis, particularly if the diagnostician is a computer pro- 
gram, The fact that a doctor does not order his exserience primarily 
in terms of attribute-disease lists may simply be evidence of the 
difficulty he encounters in attempting to deal with and maintain 
information of this form, A computer program would have less of a 
problem in this regard, and, in fact, this may be a useful structure 
to impose on the experience employed by a diagnostic program. 

While in very general terms, the functions of the program corres- 
pond to those apparently required for diagnosis, there remain cer- 
tain questions about limitations arising from their current realiza- 
tions, In a sense these are questions about the generality of the 
model, Since the program was designed to solve the model diagnostic 
problem, it is reasonable to expect that the generality of the pro- 
gram will be determined by the extent to which real diagnostic prob- 
lems can be described by the model, (Also, the appropriate statis- 
tical data must be available.) 

For example, a major difficulty in applying the program to pro- 
gram debugging is developing a proper characterization of states, 
One can see in theory how this can be accomplished, but a practical 
solution would be extremely difficult. Also, an extremely useful 


strategy in program debugging is changing the state of the program (by 


changing instructions, etc.) Here tests may very well change the 
state of the system, Because one can save a copy of the program, 
one can also use destructive testing, While one could probably 
change the model (and program) to reflect these possibilities, the 
current model does not account for them, Hence, the use of the pro- 
gram in this area is severely limited. 

Also, there may be areas in which the diagnostic experience 
may not fit the statistical model employed in this work. In these 
areas, the inference function would have to be redone for non-Bayes- 
ian inference, 

On the other hand, there seem to be a number of real problems 
which can be described by the model, including many machine failure 
and medical diagnosis problems, While the evidence is limited, the 
performance of the current diagnostic program in the areas of con- 
genital heart disease and bone tumors should not be overlooked, At 
the very least these results must be termed promising. The model on 
which the program was based and the program itself were developed 
independently of considerations of these particular diagnostic prob- 
lems, and yet the program demonstrated potential value in both areas, 
There seems reason to believe that other problems of medical diagno- 
sis will also prove susceptible to such a program, The diagnostic 
system permits the study of alternative strategies developed in the 
light of such experiments, and this, too, should ease the problems 


of increasing the extent of its capabilities. 
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Some of the difficulty in applying the program to new areas 
can be traced more directly to a lack of adequate data for an in- 
formation structure than to an inherent intractability to this ap- 
proach, If continued research yields further indications of the 
value of a computer program for diagnosis, it may well be worth the 
considerable effort required to reformulate a number of diagnostic 
problems in terms of this model or an extension of it, Certainly, 


the results of this research do not preclude this possibility. 


11. 
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Appendix 1 


Sample of an Input File 
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(STATE DO1 0.05 SO1 0.01 S02 0.10 . . . S17 0.90) 


(CLUSTR DO1 EXOR 0.05 SO6 0.07 SO7) 


(ATTRIB (SO1 S02 $03) TEST1 S04 TEST4 , . . (16 S17) TEST16) 


(TESTS TEST1 10, . . . TEST16 15.) 
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Trace of a Session with the Diagnostic Program 
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User responses = small letters 
Program responses = capital letters 


1. 


2. 


15, 


16, 


17, 


18. 


r system 

NAME OF DIAGNOSTIC AREA PLEASE 
bone tumors 

NAME OF LOSS STRUCTURE FILE 
bone losses 

INFORMATION STRUCTURE ESTABLISHED 
generate brief 

YOU OR ME 

me 

HISTORY FILE 

bone case 

CODES 

3222323 


NEW CASE 
WHAT ARE THE INITIAL ATTRIBUTES OF THE PROBLEM, Q. 


s05 s0O7 sll si4 s17 s20 not s21 


CONDITIONAL PRIOR STATE PROB 


CB 0.26 
cs 0.09 
Gc 0,62 
CF 0,02 
TRACE 0,01 


ANY IDEAS, Q. TYPE 'DONE' IF SATISFIED, 


19, 
20, 
21, 
22:6 


23, 


27, 


28, 


SET SEARCH DEPTH, THRESHOLD, AND HEURISTIC CONTROL 
1 0,10 0 


THE TEST SELECTED IS TEST43 


8343 

CONDITIONAL PRIOR STATE PROB 
cB 0,55 
cs 0.04 
GC 0.37 
CF 0.04 


THE TEST SELECTED IS TEST50 


s50 
CONDITIONAL PRIOR STATE PROB 
CB 0.21 
GC 0.78 
TRACE 0.01 


GC TENTATIVE DIAGNOSIS FOR THIS PATTERN 


CONSISTENT DIAGNOSIS FOR ALL ATTRIBUTES 
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Notes 


A, Line 7 through line 14, The user sets controls for the run, 
These controls include a history file and instructions as 
to what information is to be collected in this file during 


the run (line 13). 
B, Line 15. These are the initial attributes of the problem, 


C. Line 16. The inference function reports the current dis- 


tribution, 


D, Lines 17 and 18, The user is given the option of testing 
his hypothesis about the problem, He declines this option 


(line 18). 


E, Lines 19 and 20. Here the user sets the depth and threshold 
for the test selection function. He also chooses the stan- 


dard version of this function, 


F, Lines 21 and 22, The program selects a test and the user 


responds, This dialogue continues through line 25, 


G, Line 27 and line 28, The program makes a terminal decision 
for the pattern, This decision accounts for all attributes and 


the case is completed, 


LOL 


Appendix 3 


Listings of Diagnostic System 
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COMMON NAD 
fe ane —PLN-BUFL, BUF2,CBIT SIGNS eCPAT, ALLRAT sCPRIOR. 
1 sCTEST: ALLTST eDISEAS, STAND» FILE1,FILE2, 


3 sPATLST »STRUCT » SYMCNT op SYMLST, UNAC TDs TSTRUN,PATSTIK 
ACK, UPUNC, ARGS » PREM, CONST » SPOINT 
5 »NPRIM, CELL 


es -DIN-BURLI4AS2}, BUF2(432),STACK (20) ¢UFUNCI 20} ¢ARGS{10) — 
1» PRIM(30) NPRIM( 30) »CONST(30) eCELL{20) 


D°N VAULT(100) 
Sate ele SINITAS 0) oe Se 
SET LIST TO VAULT 


NEWVAL - ( SVALUESS,LIST.(9) LIST.(CELL(8))) 


LIST.(TSTRUN) 


see ae paeee ee wine a 


————— AISTAPATSIRKY = 2 = 
LIST. (UNACTD) 

Ceerrrereernlp SB CS BMR J 
LIST. (STRUCT) 


PRIM(2)SMINUS. 
SR LL 3a LINES... 
PRIM(4)=DIVIDE. 

HUES} 


PRIM(8)=GE. 

PREG 926 
PRIM(1O) =AND. 

re —PRINCLL AOR. neeees a 
PRIN(12) #EQV. 

i ae -PRINCA3} “NOT. = 7 
PRIMCL4)=ATTRIB. 

PAIN Ad onag§,— 
V°S NPRIM=15_SPLUSS. SMINUSS, STIMESS, SDIVIDES, 

Da enen cae: 1_ $18, $165, S508, $CE$,$68, SANDS, SORS,SEQYS, - 
2 SHOTS, SATTRIBS, SPRESS 

a —PRINT_ COMMENT—SNANE-OF DIAGNOSTIC -AREA_PLEASES 
RDLOME.« ( TEMP) 

WE @BORTGR(TEMRR 
N2=POPTOP. (TEMP) 

-_— —_—_—_ — -SE TUR. (M1 -N2} - wast Soa _ 

PRINT COMMENT SNAME OF LOSS STRUCTURE FILES 
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ROLONL.( TEMP) 


N2=POPTOP. (TEMP) 


PRINT COMMENT $INFORMATION STRUCTURE ESTABLISHED.$ 
FOP. — ———-RBLONL 1-FEMP) = 
CODE=POPTOP.( TEMP) 

—-—---— -W*R-GODEwEs SDEFINES 
LST=DEFINE. (TEMP) 

—_—________8 R-49DE-~ Ew SGENERS- $2 OR e —COBE 6 Ev § GEN G 
GENER. (TEMP) 
PRINT—CONMENT—SRETURN-FROM -GENERG «$ 

O'R CODE.E.$CLUSTRS 


PRINT OCTAL RESULTS LST 
$244 
MILIST.ETEMP) 
T*0- Ter 
Ets 


-BEAG — -MAD ee - eee - 
EXTERNAL FUNCTION (CONTRL) 
NER 


F°T P,SELECT, FANS, FANSL 


INSERT FILE COMMON 
—————— -EQUEVALENGE (2 Py P+ CEANS ANS}, (FANSL SANSI> 

R 

R pe 
R THIS FUNCTION IS THE CONTROL ROUTINE FOR THE 


R THE OFAGNOSIS. 
R 


R 

- E10 DIAG. = is 
COUNT=0 

WIS FAC 
LIST. (TEMP) 

WAR: STAND w Ew be OR a STAND oc Ew 3y -GUTRUTOLS FANDy Oy BENK) ———--————- 
R 

=Se SS AR -GET_AND- PROCESS THE INIFIAL- SYMPTONS WHICH DEF ENE.———-— ———. 
R THE PROBLEM 

—_—_S— RR cc“- 070° 0°0QQqqg—\\>—oqssss nnn nn as 
W'R CBITsEsl, P*T ILINE 

—--—--- —V¥4S—H. INE=SH/MHAT -ARE—FHE- -INT FEAL STGNS—OF -FHE-PROBLEM/#5- -- ——— 
W°R GETSYM.(TEMP}.E.0, FIN 

— ———— — WER REMPT YS CTEMP Dy FIN — —- ae 
MTLIST.(PATSTK) 

$$ 8 BB 
MTLIST.( TSTRUN) 

— ~~ — AFL ESTSCS¥MESED 7 7 
MTLIST.{ TREE) 

———--—- V4S_BLNK= $/-sH? — NEW GASE/¥/#$- ——- aos —--- 


R 
——_R PROCESS FHESE SYMP TONS FO FORM SYMPTOM BAT EE RN Sg —— 
R 
LOOP - ~ W*R-LEMPTY.tFEMP),— FO GETPAT— — . aad ay 
SYMP=POPTOP.( TEMP) 
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TEST=B0T. (SYMP) 


Ag HERE IS WHERE THE MOST SERIOUS PATTERN IS 
R RELATIVE TO THIS PATTERN. 
Rk 


QUTERUT 1 CRAT 2 0s INTERN) 

V°S INIFRM=$/,H/THE CURRENT PATTERN I1S..-/#$ 
PDUMP1.CCPAT,PATLST) 

QUT PUT 1 CRAT, 1 CLING, 

ODUMP1.(CPRIOR,CURLST) 


BB tae OUTPUT LALLPAT O¢OTENT} —-—- — 
V°S OTFMT=$/,H/OTHER PATTERNS. .0/,0$ 


R 
i a a Sa cts 6 ee 
R HERE CHECK THE CURRENT STATE PRIOR FOR A SUCCESSFUL | 
———-——— R- OLAGNOS IS OF THE CURRENT. PATTERN. —— —_ _—_ _ _ _ 


R 
—————— rea LRDR OK CURES Fe 
TL TP=ADVLER. (Re F) 
a WIR -F. Gel — ae ee 
IRARDR.(R) 
¥t9-90Fest—— — —--——_ —-— ——— — — ~~ —— _ 


OPR Polee9D 
—_—] 102 
E*L 
SSS NAMExI TSVAL.{SPNAMES, CONT..1LNKL- (CONT. (LPMER IRD} DOL) 
OUTPUT .( STAND, 1, ANSFRN,NAME} 
eye eee -V2S._ANSFRMRSH/THE CURRENT. PATTERN IS ATTRIBUTED TO /,  - ——-__ 
1 C6_//#$ 
a a 
R CHECK FOR MORE SYMPTOMS TO EXPLAIN. 
R ener ee ie _ ES eee 
Succ GOTPAT.{0) 
— —WER_ CODE E20, T'O -GETPAT _ a 
OUTPUT .( STAND, 0,0KVS} 
——-—-——-¥. LSS 2 SHACONS IS TENT_DILAGNOSIS FORALL SIGNS.f2$ 
TRALST.( TEMP) 
FAN _SOKS— a a 


R 
R- 
R 
OO76$.—_$___L. COU Fan 
WR CBITE.0.-OR.CPRIORE.2 
a NSTATE=O oe — 


RRD CONTINUE 
—— + -~ — RET Cb, WORD = 
V'S COzSCHSS 
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W°R WORD. E.SDONES 
BUF PUT ot SEAND OVUTERM 


- NSTATE=0 


V°S UTERM=SH/USER TERMINATED DIAGNOSIS OF PATTERN/#$ 
wORD=0 


OFR SORE ¢F «SOS OR SNOROEE® $$ 


Ot 


NSTATE=TRANS.( WORD, 1) 


PRINT COMMENT SNOT RECOGNIZED. 
FARR 


TRY AGAIN.S 
R 


—_—____—__A_-4 SEQDEG £15 THE FEST SELECTION ROUTE NE 2 
R 

gone —PRINE- COMMENT— $SET— DEPTH, THRESHOLD ANO-HEUR ESTIG CONTROL. $— — 
WER .NOT. LEMPTY.(ROLONL~{ TEMP )) 


--—-- —-DEP THs POP FOP tf TEMPS - 


THRESH=POPTOP. (TEMP) 

—_—__-__—__ CONF AL =POR TOPS EMA} 
Et 

SEEK NOBES]@@ ——--—-- - 
STATE=NSTATE 


WAR- STANDS GE 27 OUIPUTF 6 £2, 2cEF RM DEPTH, THRESH) 
SEQDEC.( TREE 2 Oe STATE) 


‘RDR=SEQRDR. ( ITSVAL «( SVALUESS » TREE) ) 
— ~~ —— “SFAFESSEQERS(RDRyL} 


IPSSEQLR«(RDRy EL) 
———— — ~-=-W#R WORD SNE 


=O a See e 
W'R STATE.E. SDUMMYS 
—_____—____prtegna%6——_________ 
Ore 
ee ~PREAES a 
EL 
ips ee Se 


WORD= ITSVAL. ( SPNAMES, STATE) 


Eft 


—— ——BUFPUT ss CAELT STs 35 FRADs PRE p HOR 


V°S TRMD@$H/ BEST TERMINAL DECISION AT THIS POINT IS /9€3,/ 
——-— — —— 49667 H/-tH FH EXPECTED LOSS /sF 8.208 


OUTPUT. (ALLTST +0» THEAD?) 
5 FH ttt FES F665 —____ EE 5 SH 8 
TLOOP TEST=SEQLR.~(RDRe 1) 
—— ~~ —-WYR- Fe N€e b--- — — — 7 : 
ANS#SEQLR. (ROR, I) 


~ ANS EBT FES FD 


NAME=ITSVAL. ( SPNAMES, TEST) 
a RE Sy Se Fi NAME 7 FANS by EANS—FANE A 
ves EE IMSS KE Io 9h Debt aoe k Be 20% 
+ (tee Sees FU OOF 3 
E*et 


~ QUEPUT sf ALL TST 1s SCORE s NODES) - eer 


V°S SCORE=$16-Hs DECISION NODES CONSIDERED. »//#8 
Pa a SPN et AP A UCASE TR OTP SRM EP 
R SELECT THE BEST Test : 
oa —- arrest “R ey — a ee . in” ah 
GETTST WeR CONTRL« 6.0 
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Ne 
AGAIN TOPT..CTEST,STATE) 
CKS — — —— WER STATE.NEO = 

LCOUNT=LCOUNT4LC 


POPBOT.{CELL(1)) 
MILIST.CITSVAL. ($VALUESS, TREE)) 


E*L 
DECIDE QUT BUT... STAND s-1-+DFe LTSVAL~ CL SPNAMES, STATE}} — —- —-___________. 
V°S DF=$C6,H/ TENTATIVE DECISION FOR THIS PATTERN. /#$ 
penance be ee oe OE SRLS nS 
OE 
eee ee ‘MEWGOT.( TEST, TSTRUN} — — _—_____._- 
TTEST@ITSVAL.( SEXCLUSS, TEST) 
ener WR TIEST NE. 0, NEWBOT.{TFEST,FSTRUNE — — —— ___________. 
EtL 


R TEST HAS BEEN SELECTED. NOW RUN IT. 
R- 


MTLISTo( TEMP) 

—~——— WSR CBT. 820, NEWTOR. CTEST + FEMP) _ 
OUTPUT.(CTEST» 1, TERM, ITSVAL.( SPNAMES, TEST) } 

68554 CS 
W°R LEMPTY.(CTEMP), T°O AGAIN 

TRES1 —— — S¥MP=POPRTOP.{(TEMR) 
SYMSAV.(SYMP,TEST) 


NEWBOT.( TEST, TSTRUN) 


W°R TEST.NE.O, NEWBOT.(TESTsTSTRUN) 
W9R NOT. LEMPTY.C TEMP), T'O TRESI 
T*O GETPAT 

ee V!S_CFRN=SH/DEPTH=/,12,H/_ AND THRE SH2/,F4.28$- 
V°S TERM=SH/THE TEST SELECTED IS /,CO#$ 


GENERS HAD 
EX. RNAL-FUNC TION {xp 
NER 
Fine eo _F17_RANNO,OLOP,P, RR, TESTP. were 
B'N OMARK, LEMPTY 
INSERT_FILE_ COMMON — -- =f = ws 
EQUIVALENCE (IPR,PR) 
a CEN M a 
R 
seteie o R-GENERS 1S THE SIMULATOR FOR THE. : 
& DIAGNOSTIC SYSTEM. 
LIST. (WORK) = a 
_ en Suoreeempevegny 
POPTOP. (x) 
_ __.. . ) NORUNS@FAST(CONTRED -—— — — — eae Eres 
T*Q OKTOGO 
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Ett 
—---——— -PRINT- COMMENT—$IN--FHE-GENERATOR«- -WHO—TS -CONTROLDG0$— —— — ——— 
R'T C6, ANS 
fa Sa a Bg 
C8IT=1 
+= -PRINT-GOMMENT —$00--¥OU-WISH-A-HESTORY -TO- BE- KEPT. QV$ —— — —— 
R'T C6, ANS 
H#R-ANS2 Ex SYESS 
T°O GETFIL 
9 
FILEL=0 
———-— FILE2=¢ Sane 
T°O GETING 
€%————— — — 
Ett 
a a 
R GENERATOR CONTROL HERE. SET CONTROLS. 
RR Ants erenenitas — 
cBIT=0 
ote -PREINT- COMMENT SHOW_MANY- GASES IN -FHIS RUNS Qs §- — —— 
NORUNS=POPTOP. (ROLONL. (WORK)) 


RSTAT R*T C6, ANS 


DMARK=08 


DMARK=18 
BES EAS 0 F RAINS FANS yb 
W°R DISEAS.E.0 
——— -PRENT—GOMMENT—SNOT RECOGNIZED .—TRY¥ AGAINSS-— — —————— — 
T°O RSTAT 
IPR=ITSVAL.(SPROB$, DISEAS) 
a See 
PRINT COMMENT SPLEASE SPECIFY (IN THE ORDER GIVEN) THES 
—-—---- —PRINT- GOMMENT— SFOLLOWING-CONTROL -P ARAMETERS- FOR -FHE-RUNS— — — — 
PRINT COMMENT $1. DEPTH OF THE TREE SEARCH.S 


PRINT COMMENT $3. NO. OF INITIAL SIGNS PER CASE.$ 
PRINT COMMENT $5. HEURISTIC CONTROL FOR TEST SELECTION.$ 


~————— —DEPT He POPTOP.¢ ROLONL © (WORKI  — — — — — — ~ —---_—__—____-- —_—— 
THRESH=POPTOP. (WORK) 
N59 PPR —______——————— 
NOISE=POPTOP. (WORK) 
a —CONFRE=POPTOPs (WORK) 
R 
rR 
GETFIL PRINT COMMENT SNAME HISTORY FILE.$ 


FILEZ#RJUST. (POPTOP. (WORK) 


T°O GETINE 
R- 


GEHENG PG BR ———_—_____———— 
V¢S COF=$H/FOR EACH OF THE FOLLOWING, TYPE *1° FOR A/s/y 

—— LHFGCONSOLE —TRAGEY--* 0% OTHERWISE. 706 
T*O RDINF 
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GETINE PRINT COMMENT $FOR EACH OF THE FOLLOWING, RESPONDS 
: -PRINT_COMMENT_$* 15 IF YOU WISH A CONSOLE TRACE, $—— — —______— 
PRINT COMMENT $*2* IF YOU WISH A HISTORY RECORD,$ 


RDINE PRINT COMMENT $1. CURRENT DISTRIBUTIONS 
-— —--___—- PRINT. COMMENT $2... CURRENT. PATTERNS ——— meee 
PRINT COMMENT $3. PATTERN STACKS 
Eanes -PRINT_COMMENT_-$4.- TESTS AND VALUESS — — —- 
PRINT COMMENT $5. TEST SELECTEDS 


PRINT COMMENT $7. STANDARD INFORMATIONS 

a -CPREGR=PORTOP.{ROLONL (WORK)} —— 
CPAT2POPTOP. (WORK) 

co PAT POPTOR. OWORKD —— 
ALLTST=POPTOP. (WORK) 

—_____CT§STm20RTOR. (WORK) 
SIGNS=POPTOP. (WORK? 

a STAND= POPTOP.{ WORK} — —— See ee ee 

oxTaGo PRINT COMMENT $$ 

one WERE MEO p- DUR ITE e (FILE HEAD cCPRIORSCPATSALLPAT, — — 
1 ALLTSTsCTEST, SIGNS, STAND) 


T°O DODIAG 
-- O° R_DMARK —- a . on 
T*O START 
ae EFL rn nee Buea ee a 
R 


R 
—-— Pad. — Pier ee: _ 
LIST. (GENLST) 
eee eee ROR*SEQROR. (TOR. (STRUCT). ae = 
SLooP HSHLST#SEQLR. (ROR I) 
eg 
ReSEQRDR. (HSHLST) 
-HLOOP _ __ __NEXT=SEQLR.{R, FE) a = 
WR F.Eele T*O SLOOP 
Seat! —IPRaLTSVAL.(SPROBS,NEXT) = 
P=P+PR 
—_____ MANY. LGEMLST,MEXT,Bp 
T*o HLOOP 
Petits Pac ee Onn Penne - ca sees - 
R 
i ee R WARM UP_RANNO. —— ae 
TH RIN, FOR J#1,1,J.6.20 
REN RANG 
R 
_—-- -— —R- CONTROL LOOP_FOR THE GENERATOR... 


R 
-START ———-1°H-GEND+s FOR -J= 17 le J~GeNORUNS- —— = 
OLOP=0. 


V*S COM=SH/CASE /,12#$ 


TESTP#RANNO’(X) 
-——— — — — -RDR*SEQRORAAGENLSTD = 
GLOOP DISEAS=SEQLR.(ROR, I) 
ig a ee 
OUTPUT. (STAND, 0,BUG) 
Beh see VtS BUGSSH/BUG IN GENLST/e$ -- — — ee 
CHNCOM. (0) 


eR ee ane RN TS RE SE A A A A A 
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OfE 


— = EPRESEQERSERORG ER --- 7 
WIR OLOP.LE.TESTP.AND.TESTP.L-PR, T*O GOTIT 
—_—_—_—__—_——atpP=PR 
TO GLOOP 
at €41- Reiss ate e 7 ioe ae a4 8 ee ee cay heen iat 
R 
a a R Secs gis fmtea ae —— ae peer eae iat -— 
GOTIT OUTPUT. (STAND, 3, HEAD + J, ITSVAL «( SPNAMES,DISEAS) »PR-OLOP) 
DODIAG DIAG. (CONTRL) 


a WP R-EBET 6 Ee FE — pc oe ee aes jee Res 
PRINT COMMENT SANOTHER- aes 
eo ees R'T-C6, ANS —— -—— ct races ate mete oy 
WIR ANS.EWSYES$, T'O DODIAG. 


Eft 
GEND-——- CONF ENUE ae Mis ole Sapte Due Nee 
OUTPUT. (STAND, 0, TFRM) 
“— ~~ ARALSTSAGENESFP oe Sofa 
V°S TFERM=$//,H/RUN COMPLETED./#$ 
cca te Smaak a Se 2 OR 
IRALST.( WORK) 
RN apaokies oo re? ad ale nefectied cece te 
R 
R és i —— - a eas cee feast 
V°S T1L=$11*$ 
415. €6=46605— 
V*'S HEAD=$H/SWITCHES FOR THIS RUN/»/, 
-—--  --E F#HEPR HORS, By 18 p SHOPAT=y149 Sy FHALLPAT=,— a aa 
2 TlelS,THALLTST=,11,15S, SHGTEST =a heel Se 6HSIGNS=, 
a 3- E15 1S sGHSTAND=s Hy / 4 - ae 
V"'S HEADL=$//, Vi/dandcuductksecawsnaceeusteeceseceeus: 


ty 6y 2H — 


2 F3.292H)o/*$ 
= oe = —-- —- 


GETSYM MAD 
oS HERFERNAL FUNCTION (ESF }-— —— -a = ee 
NPR 
——- FAT TESTPyPRePOLDsPNEW,PEUyRANNG— ——— 
INSERT FILE COMMON 
$8 NEM PY AM EST SP EST 
R 
—--—R-FHES FUNCT-EON- HANDLES -ALL—THE SIGN RETRIEVAL —-- ao ne 
R ACTIVITY FOR THE DISEASE GENERATOR AND THE 
—R-BEAGNOSTIC PROGRAM, © —— -——-- — 
R 
$$ -€48-6E FS 
RET=1 
£5. LISTS(WORK) -——  ——- - See Jy Ss 
W°'R CBIT.E.1 
WIR SIGNS «Gel — a eo 
DS=2 
Ee 
DS=0 
aor er -E#E = te tes ode ait 
R 
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R HERE THE USER IS IN CONTROL. SIMPLY RETRIEVE 
eee —R_ THE NEXT SYMPTOM FROM HIM ( WITH TRANSLATION oe 
R AND CHECKING). RECORD SYMPTOM AS CALLED FOR. 
a 
OUTPUT.(DS,0,FIRST) 
Be 3. -ROLONL .1WORK) = aes : 
Loop WSR NOT. LEMPTY. (WORK? 
Mee at ae -NAME=POPTOP.( WORK) a 
W'R NAME.E. SNOT $.OR.NAME.E. SNOS 
——_____ ML =poprop.twork) 
W°R NAMTST.(NL) 


ae See R=SEQROR. (NLP -— ——__— — dine a 
SL NL=SEQLR.(RoF) 
Rn 
STRANS.(0) 
eh SS a A ree oad OE ge Meee ted gee el alg ey A ee Sa Wh a Et 
EL 


INTERNAL FUNCTION (0M) 


WORD=TRANS.(NL,2) 
——— — — — -WER-WORD. Es Oy F*O- ERRMRK— ae Sr 
NEWTOP.(-WORD,LST) 
——— —DUTPUT «FDS ote NRMe NLD 
FPN 
$$$ SSeS 
O*R NAME.E. SNORMALS 
ae Re SEQRDR. CPOPTOP 21 WORK)? 
Te NEXT=SEQLR.(R»F) 
saa ee WER Fa NES Lb 
OUTPUT.(DSs LyNTpNEXT) 


RL=SEQRDR. ( ITSVAL. ( SMEMBERS , TRANS. ( NEXT» 3))) 
20 ener _— —-S¥MPaSEQLR{RIZFL) —_ 
WR FLLE.d, T'O TL 
NEWDOP. (-SYMP,LST) 
TO Thi 
ee ee as ee 
O'E 
get a Riese ——WORD=TRANS.{NAME,2) is esl 28 
W'R WORD.E.0, T'O ERRMRK 
Sere SF _NEWTOP.(WORDeLST) 
OUTPUT..(DS_1,POS,NAME) 
aaa 
T*D LooP 
= ey = ars eee = pe ity =s 
R 
See R-WHEN- THE CURRENT LEST 3S EMPTY, INITIAL SYMPTOMS = — 
R MUST BE GENERATED. 
een Sean ar eC a ee a a a 
O'R CURLST.E.0 
eee eee ere . OUTPUT {SIGNS sO, INIFRM) Peas as 
MANY. (LISTs (TEMP) »DISEAS, 1-0) 
een, eee COUNT2RELTST.{ WORK, TEMP) — — Beets 
TRALST. (TEMP) 
eee Caen ne ee Eee 
R HERE THE INITIAL TESTS ARE CHOSEN AT RANDOM 
PESTO OBER THE-IRETTAL -SERPTONS © = ee 
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SwiTCH=0 
T*H-FELOGPy FOR- dete lvdsGeNINHS 
WR COUNT.E.0, T°O OUT 


GETL TEST=SEQLR.(RDReT) 
Keke ———— 
W°R KoLeKTH, T*O GET 
$B 
NEWTOP.(TESTeLST) 
a SYMGENStEST? aa ~ 
REMOVE..{LPNTR.{ROR)) 
WER-FOP AHS F650) SW HEHEL 
TGLOOP CONTINUE 


GUTPUT.(STAND,0,NOS) 
Fag 


Rk 
R HERE A RESPONSE TO A PARTICULAR TEST IS 


R 
SYMGEN.(LST) 
E*t 
R 
BG KB RAE SFr J 
FON RET 
EARMAK——— -QUTPUT of STAND sO, ERRI- 
T*o Loop 


R 
R THIS FUNCTION SELECTS A RESPONSE AT. RANDOM 


R THE KNOWN DISEASE *DISEAS*. 
re 

INTERNAL FUNCTION (X) 

£40--SYMGEN. = eee 
TEST=POPTOP.{LST) 


SPTEST=18 


vate ; 
TESTP=RANNO.(X) 

$$$ 8808 $$ 
R=SEQRDR. ( ITSVAL. ( SMEMBERS, TEST)? 


GLOGP — — — -NEXT=SEQER. ER, S} - 2 
WR S-E-1 
WR-SPTEST, FAN = - 
GLooPL NEXT#SEQLR.(R2S) 


HERR 58d i 
NAME=ITSVAL.( SPMAMES,NEXT) 


aa -¥°9-GLGOPL ~ - — 


W°R LOC.E.0, T'O GLOOP 
Stee i PROP Ed o NEXT, CONF. Hi NKR (CONF o{ L061) 413} 
PNEW=POLD+PR 
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W'R POLD.L.TESTP.AND. TESTP.L.PNEW 

Sa -NAME=1TSVAL -1-SPNAME Ss NEXT) ———--- 
OUTPUT .{ SIGNS» 1,POS,NAME) 

WE WTOP 46M T AS Fp 
W'R SPTEST 

POLD=0~ es 

T*O GLOOP 
AL a 
FN 

gt ee 
POLD=PNEW 


Seen -V1S FIRSTR$H/USER RESPONSE /,COas_ 
V°S NOS=$H/INITIAL SIGNS ARE ALL *NORMAL® SIGNS./#$ 


wh ae ees V2S ERR=SH/SIGN. NOT RECOGNIZED. IGNORED. /e$ —§ = 
V°S INIFRM@$H/THE INITIAL SIGNS OF THE PROBLEM ARE/eS 


V°S POS*SH/OBSERVED SIGN °/,C6_92H*.#$ 
EAN - 


POUMP = MAD 
eo EXTERNAL_FUNCT ION {HARK} 2 
NIR 
FAT_WGT- 
EQUIVALENCE (1WGT,WGT) 
—_____INSERE BILE COMMON 
E*O OUMPP. — 
ones eat _WER_ALLPATE20¢ EEN 
OUTPUT. (ALLPAT» 0, BLANK) 
See -RaSEQROR. (PATSTK} 


—WER_COUNT -Eo1, OUTPUT. {ALLPAT.0,ONLY) “ 
OUTPUT. {ALLPAT» 0, BLANK 
(teas whats wart =~ deers gh dhebngsi a = 
EL 
——_________ COUNT =COUNT43-—___— 
WIR NEXT.L.0, T*O LOOP 
— ~~ —— — WER _NEME.E. CURLST.. ANDO. CRATES ALLPAT:3°0_LOOP ae 
IWGT=I TSVAL. ( SWEIGHT$, NEXT) 
(26 See —POUNPL {ALLPAT I ESVAL ~LSSYMPSS NEXT) > . 
QUTPUT.( ALLPAT, Ly» CLINE, WGT) 
DUNE BAT NERF} 


T°O tooPp 
R : 


R 
a VES BLANK 26205 _ asst = _ 
V'S ONLY=SH/CURRENT PATTERN IS THE ONLY ONE./#$ 
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DUMP1 MAD 
FERN AL FUNG FEEON EM ARE SF 
NER 
FLT -p,pTor —- z Aue sees pti iy 
EQUIVALENCE (1P_P) 
-ENSERT FILE COMMON é Sac = Lae ae 
—£°O PDUMPL. 


FIN 
CNT=0 
: R=SEQRORS{LST} ane “= oo fs aS 
Loop SYMP=SEQLR«(ReF) 
-—- WIR Feed - mee soa 22 = 
WIR CNT.G.0¢ OUTPUT. {MARK sCNT,SYLIN, SARRAYS) 
Qu Pt AR Ry Oy BEAN 
FEN 
Mest has - -E9, : . poreras edyed ec caf 
WIR SYMP.L.O 
- ~- CNTSCNT4L ce a5 oe 
STACK(CNT)=$NOT $ 


ae 
CNT=CNT#1 
STACK{IENT POETS VAL « ( SPNAMES, SYMP} .V2$000,008-- " po! ete 
W'R CNT.G.17 
GUT PUT . (MARKY ENT S¥L EN? SARRAYS) eae Sou _ 
CNT=0 
44 
T*O LOOP 
R 
R 


- €°O DDUMPLa. - seh. *eahans 
W'R MARK.E.O, FIN 


—_——____P 880+ 
R=SEQRDR.«(LST) 
QUT PUT « (MARK, 0, DL ENE) ae 
LoopL STATE=SEQLR.(RyF) 
WR FoEs1 ; aes 


OUTPUT. (MARK, 2eLINE, TRACE, 1.-PTOT) 
a IM RI 9B BAN 
Fn 
Eft 
IP=SEQLR.IR,F) 
WR PoLeleE-2, FG LOOGPL . 
OUTPUT .{ MARK, 2,LINEs ITSVAL.~ (SPNAME$, STATE), IP) 


T'o LOOPL 
R 
R 
VS BLANK=$/*$ 
V'S TRACE=$TRACES 
a HS SHEEN SEBO HS— 
V*S OLINE=$H/CONDITIONAL PRIOR STATE PROB/y/#$ 
V"'S LENE=$20SyCOsF 4-205 
E'N 


OUTPUT MAD . ae 2 - 
EXTERNAL FUNCTION (MARK, NARGS, FMT, Al,A2,A3) 
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NIR 
-——-. -- INSERT FILE -COMMON 
E'O OUTPUT. 
WR Ade Be SARR AY $9—F2G-DO MARK 
STACK(L) SAL 
Se tees -STACK{ 2)=A2 an = 


00(3) CONTINUE 


OWRITE.CFILEL, FMT) 


DWRITE. (FILELS FMT, STACK(1}00sSTACK(NARGS) ) 
ware apa a = 
T*G DO(MARK-2) 
DOLL) _wiR-NARGS.g.9 
P°T FMT 


P°'T FMT, STACK(1).. STACK (NARGS) 
EAL 


SELECT MAD 
—________EXTFERNAL_FUNCTION COUMMY) 
R 
A. ne RL THES FUNCT LON EXAMINES ALL THE PATTERNS INTHE. 
R PATTERN STACK. IT RETURNS THE NAME OF THE PATTERN 
eee _R_WHICH-HAS MINIMUM EXPECTED LOSS AS *PATLST#. 
R THE DISTRIBUTION CORRESPONDING TO THIS PATTERN BECOMES 


R 
o ee NER ats se 
INSERT FILE COMMON 
i FLT WEIGHT,WGT »PSAVE. Qe 
E'O SELECT. 
pe yee 
FIRMUP.(PATSTK) 
_- . -— — RDR#SEQRORSIPATSTK) gt eh de Selig ee 
SAVLST@CURLST 
TLODP___NEXT=SEQLR.{RDReI} sgh eee eS Rad Sel rth a 
WR TeEol 
_————— 
R UPDATE THE TREE 

ee elt oS RE 


MTLIST. (TREE) 

~~ = — ———-— NEWVAL A SVALUES$ LIST (9), FREE} — — a 
NEWVAL . ( SPRIORS,CURLST, TREE} 

——__________W1R_SAVLST NE CURLS Te MELISTOICELLEDD) 
F'N PSAVE 

Sane _. -ESb > Cpa as Se ae pS cpa a Det Ea Saas wh es 
WGT2WEIGHT. (NEXT) 

aoe os — -NEWVAL ASHE IGHTS WGT, NEXT) = 
W'R NEXT.L.0, T*O TLOOP 

{——__> WER -UGT Gs pS aye 9 = 

PSAVE=WGT 
wen. CURESTSNEXT.-- = een a 
PAFLST=ITSVAL.« (SSYMPSS NEXT) 
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Ee. 
+¥*9-TtaePp- = 


R 

sR 
R THIS FUNCTION UPDATES THE UNACCOUNTED~ 

eee R-FOR-LIST AFTER & -SUCGESSFUL—DIAGNOSTS OF —A-PATTERNS ——— — ——- 
R IT CONTROLS THE FORMATION OF NEW PATTERNS FROM THE 

a R- SYMPTOMS -REMAENING -ON: THE -UNACTO-LIST« . 


R 

$5 80.0 AF 
CoDE=1 

‘——--- -RBR*SEQRBR= (UNACTD} — ——— — ———— ——— - -—-— -~-—-———~——_- 

Loop SYMP=SEQLR»(RORg I) 

GHEGK——— -WtR- Be Evy F410 -PRUNE-——--— — —~——-——— ———~~---——-- 


LOC=MEMBER. (SYMP,PATLST,0) 
reece tar Wy ie Sn; nen 
W°R SYMP.G.0, CODE=0 
— F*9-+--9ee— — — — — — nr eee ee 
O°E 
—----~-- -ADD@LPNTRARDRI- — —— —— —-— ———— -— —— ~—- 
SYMP=SEQLR-(ROReT) 
$$ REM A 
T*O CHECK 


Ett 
R 


R- 

PRUNE WR COOE.E.1, FIN 

RB Re SE GR BR EPA ES Fe} 
LIST. (TEMP) 

LOOP 1 ——— NULST=SEQERs(RDRed 4 — — ee 
WR LeEol 

- ¥°9-RESTGR - 
OfR NULST.L.O 

~ NEWB OF NUE SF FEM 
O*R NULST.E.CURLST 

--—---—- -NEWBOT 3 (-NULST p TEMP} — -- - 
Et 
720-LOgPL 


ee ee 
RESTOR RDR=SEQRDR. (UNACTD) 

——-—--— MELIST wf PATS EK} - - - 

Loop2 SYMP=SEQLR.(RDRy I) 


INLSTR. (TEMP »PATSTK) 
Sn EEE 
FIN 
plete -€4#4-----—--- tas see Rgee oek pas viele use ad acco eae bes atk ae & Sere a — —_ = 
W'R SYMP.G.0, PATFRM. (SYMP) 
a + -F89-400P2 es 
E'N 


‘PATFRM- —-MAD------- == --— - 
EXTERNAL FUNCTION (SYMP) 


R THIS FUNCTION FORMS ALL THE DISTINCT PATTERNS 
— —R FGR-A-GEVEN SYMPTOM, —*SY¥MP ¢. IF PROCESSES -a-e 
R ALL PATTERNS SO FORMED AGAINST THE CURRENT 
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R PATTERN STACK. If THE PATTERN IS A NEW ONE, IT 
pee —R-1S RETAINED. OTHERWISE IS DISCARDED.-— —__—.. -- — 
R 

i _ 
B'N SUBSET 
_-— INSERT_FILE COMMON — Bigs eat = : 3 
F°T Pp, UPOL 
——-E1.0-PATFRM. eS res ASE yg: Se Set 
MEMLST=ITSVAL.{ SMEMBERS, SYMP) 
a Se 
R PROCESS THE SYMPTOM PATTERN FOR EACH STATE ON THE 
Bee Soy A MEMBER LISE'GE Saxe r~ -- 23 


25> = = pppcee cae aNenL Se) : = ee : 

Loop STATE®SEQLR.(RDR»T) 

—____—-—_______WER_] Ee FON eS 
. 


——— | ~-R-CHECK-FOR THIS STATE_IN THE CURRENT PATTERN STACK. © .-——__—- 
R IF IT IS THERE THEN ITS SYMPTOM PATTERN MUST ALSO 

_--———-R BE THERE, AND IT SHOULD -8E IGNORED. = — jon eS: : 
R 


W°R MEMBER. (STATE, PATSTKs1).NE.0, T'Q LOOP 
2 R- ye eee sath a : 
R STATE NOT FOUND IN PATTERN STACK. SYMPTOM PATTERN 
Fp Se R-FOR THIS STATE MAY_BE A NEW PATTERN. — a: 
R GET THE *PARTIAL SYMPTOM PATTERN! FOR THIS 
AA EIS CARS OS 


— INSECT -LUNACTO, STATE, LIST.ATEMD)) A 

R IS THIS PARTIAL PATTERN A SUBSET OF AN EXISTING PATTERN.Q. 
Sore --- R=SEQRORe APATS TK} — — -- —— Sage 
cLoop NEXT=SEQLR.(RsF) 


WR SUBSET. (TEMP, ITSVAL. (SSYMPSS»NEXTI) 
-- ee _ERALSTO CI EMR DL Sie ee, Voy RO SS aa =: 
T'O LOOP 
_ = OtE_ ese = es ~ = _ = 
Tto cLooP 


Ett 
a pets Se as Rae . = pa charter = _ 
R "TEMP* NOW CONTAINS THE PARTIAL SYMPTOM 
___.. --_R PATTERN. CREATE THE STATE_PRIOR FOR THIS PATTERN. ice Ste 
R AND ADD IT TO THE PATTERN STACK. 
R 


NULST=CONT. (NEWBOT.(LIST.(9) sPATSTKI+1) 
_— —_NEWVAL .C-$SYMPS$, TEMP yNULST} ae, Ze 427d 
IRDR=SEQRDR« (MEMLST) 
INLOOP. ——-STATEL*SEQLR-LIROR, II) - Bees sles Zone 
WIR TL.Esl, TO PROC 
—___-___SEQLRLIROR, HY 
WR MEMBER. (STATEL,»PATSTK,1)-NE.O, T'O INLOOP 
re MANY. (NULST,STATEL, LTSVAL« { $PROBS, STATEL)) ates eee cet 
Ta INLOOP 
Poe Seearenne Soe, MES Sam 
R "NULST® NOW CONTAINS THE STATES AND A PRIORI 


R UPDATE THIS PRIOR BASED ON THE VRE TONE IN "TEMPt. 
R : ener 


PROC P=0. 
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PROR=SEQRDR. ( TEMP} 

PLOOP ~ —— -S¥HPL=SEQERS(PRORY PHI — wea 
WER PILNESL 

ESP AE og 

T*O PLOOP 
ae ao) ey £41. ome See! + ae iitieshs dake 2 Se ity, 

IRALST.( TEMP) 

~~ -¥90-4086P --—- 
E'N 


UPB-—-MAD- —------——-- -- aes beste ois ~- 
EXTERNAL FUNCTION (SYMP) 
qo 
R THIS FUNCTION SUPERVISES THE UPDATING 
—  ———-R OF-THE PATTERN STACK-—GIVEN -THE-NEW *S¥MP 4. = 
R EACH OF THE STATE PRIOR LISTS IN THE STACK 
——--R 1S-UPBATED-(PROVEDED THAT—THE *S¥MP4 1S ———— 
R RELEVANT TO SOME STATE IN THE LIST). 
——_—__R_- WHENEVER THE PROBABILITY OFA PATTERN GOES ———— 
R TO ZERO, THE PATTERN IS DELETED FROM THE 
a R-PATTERN STACK = — : ——— = 
R 
NE Rr me sini ps ehh ett wibe By Pate _ 
B'N LEMPTY, RELEV 
INSERT 


F°'t P,UuPDL 
<a — €10- UPB. - - = od - = 
NEWBOT.{SYMP,UNACTD) 
SS aie oe -RBR=SEQRBR«EPATS IK} ———' _ eee = cae 
Loop STLST2.ABS~SEQLR.~(RDR,I) 
GHEGK —_W*#R Ta Ee by F428 FINES HH 
R 


——. R CHEGK-THE RELEVANGE. OF THE -S¥MPTOM TO-...— —-- oe 
R THE PRIOR IN STLST. 
a RS cee tee nets a 7 ee 
W'R .NOT. RELEV.(SYMP,STLST) 


T*O LOOP 
ee Eft Seals pee ae et co ig See he Se Seale 
R 
----- = R-FHE SYMPTOM IS RELEVANT. USE ET-10 —-. Bids Ge GSS 
R UPDATE THE PRIOR. 
sy 


P=UPD1.{SYMP,STLST,STLST) 

WER-Ps Es -— ao ean 
WIR STLST.L 00, UNDO.(STLST) 
ADD@LPNTR. {RORD -- ee a 
STLST=SEQLR~(ROR» 1) 


PLOOP WER LEMPTY.(SYMPS), T*O CHECK 

a TSYMP=POPTOP.(SY¥YMPS}-— — oe 
W°R TSYMP.G.0, PATFRM.{TSYMP) 
T'O PLoOP-- -——-. . : stipes 


——_—________NEWVAL 4 SPROBS2PYSTES TD 
NEWBOT. (SYMP, ITSVAL.{$SYMPS$,STLST)) 
——- ¥*9 Looe - ee 

EPL 
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R 
R_HERE THE SYMPTOMIS- PROCESSED BY (pATERME = 
R TO SEE IF ANY NEW PATTERNS CAN BE FORMED. 

FINISH W°R SYMP.G.0, PATFRM. (SYMP) 

= = SPE! oe ee estes Seve: 

E'N 
—UPOL = NAD 2S oo ee Se 


THIS FUNCTION UPDATES THE STATE PRIOR 


R SYMPTOM "SYMP*. THE SIGN OF *SYMP* DENOTES 

eee R_ THE -PRESENCE_OR ABSENCE -OF -*S¥Mme, st 
R *LST2* IS WHERE THE UPDATED PRIOR IS STORED. 

Sees a nc ee es es 


——_________-¥#S_ERSIal.Eeg 
INSERT FILE COMMON 
——— — — —-F LTR, RES EPST PRe PROB 25 - 
EQUIVALENCE (1PROB,PROB) 
awoeses BIEN. SAME =i 
E*O UPD1. 
Wt RAST BLS 92 
SAME=18 
OLE Mais ci! re — 
SAME=08 
s uA pees os 
P=0. ; 
——___MEMLST=1TSVAL.(SMEMBGRS, SYMP) 
R 
2a SS R_PROCESS EACH STATE ON THE MEMBER LIST OF _*S¥MP*. 


————— ~-RORMSEQROR-LLS TIL) ee 
Loop STATE@SEQLR.(RDR, I) 
CHECK. __ wee FE.) 
W°R P.LEPSI, F°N 0. 
—— + ~RDRALROROV.{LST2) . - - 
AGAIN TPROB=ADVLER. (ROR, I) 
——S WERT. Gol, EM 2. = - 
ADD=LPNTR.(ROR) 
SUBST 1A OROB 42s ADO) 
T*O AGAIN 
mn rare Parner ae ae aS 
IPROB=SEQLR.(RORe 1) 
----- +L GC=MEMBER.{STATEsMEMLST 0} - - 
WR LOC.E.0 
Re 
OrE 
eg ae eae ape PREPLJ 21 SYMP, CONT.{LNKR.{CONTF(LOC) }41)) - 
EE 
ene WER_SYMPLL WD = eg ns ent : 
PROB=PROB#(1.-PR) 
SERRE ce ae 
PROB=PROBeEPR 
rt te «er Soit pBace Et gh Ret Napa, Be toy slab en? Reins eet cee seit Sete, heme’, GR aerate. \ ae smor ete! wfc hbeSy AES oes pes ety 
R 
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R CHECK FOR *ZERO* POSTERIOR FOR THIS 
+ —R- STATES -EF-ZEROy DELETE 17 FROM -FHE- LIST s 
R 
wR a Py EBS RR 
P=P+PROB 
—— WAR-SAME 
ADD=LPNTR.(RDR) 


R 

SERA Pe a yh 
ADO=LPNTR. (ROR) 

= ABDtT=t NKE (CONT. £409) — — — —-—- ——- --— ——--—-—-—- —- 
STATE*SEQLUR. (ROR, I) 


REMOVE .( ADDL) 
——————____F8. §-§ HEG K__$_——————————————— 
E°N 


——P ed: 4 
EXTERNAL FUNCTION (SYMPeCLUSTR) 
NAR _ 
INSERT FILE COMMON 


FT PL,P2 
—_____EQUIVALENGE 494, 2P Eb, ¢p2,- Ppa —— 
E’O PIJ. 
Rg 


R THIS FUNCTION OBTAINS THE. PROBABILITY OF SYMPTOM 
eee ener R- AS¥MP-4.— I CLUS TRS -IS- EITHER -FHIS PROBABLE TY-oR — — — — — — 
R THE NAME OF A CLUSTER WHICH CONTAINS *SYMP*. 
aN ee eee 
WR NAMTST.{CLUSTRI» FON CLUSTR 
a a LEST © ATEMP) ay : 
LIST. (OPSTCK) 
~ +» — — ——RORSSEQRORWAGLUSFRI— —— — - 
Loop NEXT#SEQLR. (ROR, 2) 
HR a 


R 
—-—--- R- SHOULD NEVER-GET HERE — - - a ----- a: 
R 


FAN--ts ae eo gee 
O'R NEXT.E.LPAREN 
aork NEXT .EsRPAREN 
: R R END OF A A TRIPLE, PROCESS OPERATOR AGAINST ‘TEMPS 
Sete ant " [PL=POPTOP. (TEMP) » Se 


W°R LEMPTY.(OPSTCK) 


R END oF THE EVALUATION 
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Bsn 4 IRALST.( TEMP}. e re Sap lus corte, 2 
IRALST~(OPSTCK) 
_—___£1N-PL 
EL 
-. . EP2=POPTOP.( TEMP) anwar $0523 tee ee 
SECOND=POPTOP. ( TEMP) 
= --— OPER=POPTOP.(OPSTCK) ——- ewes ie Se iol. yas 


-——WIR- OPERSESSORS— = -- 2.4 Sesty fee 
WER FIRST.E.1.0R.SECOND.E.1 
oe PP LEP? —- Z NS Seas, . arenes . 
BMARK=1 


=e a OLE oe fee ris wise . oper Bi Bein, fect 
BMARK=FIRST+SECOND 


OfeE 
oo -PLSEPL4P2 aH a A a ee --- 
Eft 

+. 

R 

R. CHECK.FOR. AN OPERATOR HERE —— ee 

R 

OR NEXT Es S$OR$S.GRNEXTE~SEXORS — —- 

NEWTOP. (NEXT sOPSTCK) 

—________—_—___140 Loop 

R 

R PROCESS. SUBCLUSTER HERE — = ies 

R 

-O°E 7 . —_ ss pane 

BMARK=INTERP.(NEXT)} 

W!R SYMCNT.£.0 
Pl=0. 
BMARK=0 : ot ee 

Of*R BMARK.E.O 
P1lz0. 

O'R MEMBER.(SYMPsNEXT,0O).NE.O 


IPL=ITSVAL.{$PROBS,NEXT) 
Ore 
Pl=l. 
EL 
Ore 
Pi=0. 
Eth 
etl ne 
MANY. (TEMP, P1,8MARK) 
T'O Loop 
V'S LPAREN=8($ 
—_____¥$_RPAREN=$)$ 
E'N 
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NSCOMP MAD 


NER 
a EST PY UPDE a2 nae = ae 
E'O NSCOMP. 
~——— + -ReSEGRDRs CETS¥AL s(SMEMBERSHTESTIR (= 
LooP SYMP=SEQLR«(RF) 
#4 NEw —_§$ 
P=UPD1.(-SYMP,PRIOR,LST) 
a WR -PaGsGe7 F400 LEGP- > a 
Eft 
—- ee - = ee Hae ee Be 
E'N 


DEFINE — -MAD— tae, Gets Fg, PS bees A te 
EXTERNAL FUNCTION (TEMP) 
NR 
B'N LEMPTY, OPER 
—— —-INSERT FILE COMMON -- ~~. Ss eS Se 
E'O DEFINE. 
~——-t$T<9.- ——~— 20 Sass ae 
LIST. (RDRSTK) 
NAME =POPFOPLtFEMPY 
NUM=UFUNC (0) 
-——--  TEH-LOOK, FOR J=by2sdeGeNUM — Pete, ey aS 
W'R UFUNC(J).NENAME, T*O LOOK 
-——~- WER UFUNG(J4+1}.NEWO io= a 
PRINT COMMENT $RELATION ALREADY DEFINED. REPLACE.Q.$ 
RAT $6625, ANS- 
WIR ANS.NE.SYES$, T'O DONE 
a LST=L-EST~ (UFUNC (+1) }—— ae . gaa Bay «nfo 0 
T*O START 
parr ee Ett Sa cae ane Z oes Sixt seat = eens 
Look CONTINUE 
—————-NUM= NU M42 
UFUNC ( NUM} =NAME 
——EST=LIST« (UFUNC(NUM#+1)) eee ae San -- 
PRINT GCTAL RESULTS LST 
ae UFUNC {0} =NUM —— - 
START PCOUNT=0 
~OPERALS 
ARGLST=POPTOP.( TEMP) 
oo RBR*SEQROR. (TEMP? — As 
Loop ELEM=SEQLR.(RDReT) 
-—- WER ToEe2 
PCOUNT=PCOUNT~1 
HREM REA SHE- 
W'R PCOUNT.G.0 
- PRINT COMMENT. SNOT WELL FORMED. TRY AGAIN. S$ ae 
UFUNC(J+1)=0 
: ett 
T*O DONE 
ore 
NEWBOT.($)$, LST) 
ROR=POPTOP. (RDRSTK) 
T*o Loop 


212 


Ett 
-— OLR-1.b.0 apace were ear 
W'R OPER 
OPER=08 
W'R ELEM.E.$QUOTES 
-. — —. --ELEM=SEQLR. (RDR ALD ae = 
T'O COEF 
ye Se Et paar See fete ed 
CODE=ARGTST.(0) 
Wk CODE NEO, ELEM=CODE.V.S4K1Q 
NEWBOT.(ELEM,LST) 
Ore. ben RS a eet nace 
CODE=ARGTST.(0} 
W'R CODE.NE.O —— — : ee sah 
ELEM=CODE.V.54K10 
_—__________________ MEWBOT.(ELEM,LST) 
O'eE 
COEF... ... —— —— ENUM=CONST{0) —-- Seep “ste 
CNUM=CNUM4+1 
CONST(CNUMD=ELEM ae Shea Ss 
CONST(0)=CNUM 
NW BOT NUM ES 
EL 
iieeE be He | EYL 2 ; a3! oes 
Ore 
SE. Ope RS pia a ae ‘ 
NEWTOP. (RDR,RDRSTK) 


NEWBOT.($($,LST) 
PCOUNT=PCOUNT +1 er ie 2 oe 
Eth 
joe EO Leope st Se —_ = 
R 
R 


INTERNAL FUNCTION (OUMMY) 
R. : 
—'O ARGTST. 
~ -ARDR=SEQRDR.« (ARGLST) 3 ae ne — eet 
ACOUNT=1 
ALOQOP __ ATEMP=SEQLR.{(ARDR-AL) 
W'R AIJE.1, FIN O 
“WER ATEMPESELEM, FIN ACOUNT 7 = 
ACOUNT=ACOUNT +41 
T'G ALOOP 
E tN 


DONE IRALST.(RDORSTK) 
Fen LST 
E'N 


CLUSTR MAD 
EXTERNAL FUNCTION (LST) coe 
NFR 
BAN LEMP TY 
F'tT Py PSAVE 
EQUIVALENCE CIPsP} 
E*O CLUSTR. 
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LIST. (TEMP) 
~~ STATES TRANS {POP FOPSHE ST} yh} —-— 9 = 
WIR STATEE.0, TO ERR 


NEWVAL. USRELAT Ss SCLUSTRS9NULST) 
=e RDR=SEQROAELST) ----—-— ee: —---- 
NEWBOT. (LPAREN,NULST) 
—--— -§UB20: -—— -——-—- ----— — tte ee ee 
Loop IP=SEQLR. (ROR, IT) 
$$ SSS 
NEWBOT. (RPAREN»NULST) 
a WER EMPEY. (TEMP a 
TRALST.( TEMP} 


ROR=SEQROR.( IP) 
on NEWBOTF st PAREN NULS TY coe a soe 
Eft 
-—-OAR PPe&s $ORSsORSEP SEs SEXORS- ——  -——— -- 
NEWBOT»(IP,NULST) 
BE 
SUB=1 
PSAVEsp ———.. SPE SOS 


Et 
T'g-LogP ——— seen aera oo 


R 
R 
INTERNAL FUNCTION (SLST) 
£*0 DOL. —- — eee 
SUBLST=CONT. (NEWBOT.(LIST.(9) pNULST) 41) 
NEWVAL « { SPROBS »PSAVEsSUBLST) sep SSS 
NEWVAL .( SRELATS, POPTOP.(SLST), SUBLST) 
——~_____DRDA=SEQRDRWESL57} 
pLooP NEXT=SEQLR.(DORDR,DI) 
W'R-DEEels FIN —-- to ee ——--- 
NEXTSTRANS « (NEXT? 2) 
oe WIR NEXT eEeOr T'O-ERR-— -— - oe 
LOC=MEMBER. (NEXT ,STATE,0) 
REMOVES +L 8E+— 
LOC=MEMBER» ( STATE» ITSVAL = ( SHEMBERS »NEXT} » 0) 
ADD=LNKRs (CONT «{LOC)} Sees ee 
SUBST. (NULST, ADD) 
NEWBGT {NEXT SUBLST} fp aae eye 
TO DLOOP 


DR 


ERR PRINT COMMENT SERROR IN FORMATS 
FENe-i ate 
V'S LPAREN=$($ 

AS RPARENES HS 
E*N 
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INTERP MAD 
—_________EXFERNAL_FUNCTION -4CLUSTRY 
NFR 
tee INSERT FILE COMMON - : ae Gc) : = 
E'o INTERP. 
S¥MCNT20-—- _— = eer ere — 
BASE=0 
_—______SROINTag 
STACK(0}=0 
-——. —-RDR=SEQRDOR~ CLUS TR} ——-—- ee ie yeas 
Loop ELEM=SEQLR.(ROR, 1) 
bo eda, WER ICES 2 Z Bos ce A 3. 
PUSH. (ITSVAL.( SRELATS,CLUSTR) ) 
a Oe e AL tO) 
WIR STACK(0).E.S#INC#S 
- ct - -STACKIOI=1 — —-—- a ge oe 
O'R CODE.L.0.OR-STACK(0)-E. SFALSES. 
Sei oo 2 STACK P ale ee Ain Fe Ds eee 
E'L 
NE STAC CLS 
O'E 
ae -—— -PUSHA4ELEM) — — —- Sie ae tag te Cg - cae 
T'a Loop 
Ste eS epee eae eee Le a : oo. i Jee 
E'N 


—-EVAL—... MAD ee ee - a 
EXTERNAL FUNCTION (DUMMY) 
—____—___At Ra 
INSERT FILE COMMON 
SS Gated ENO EVAL oon Bis bs OY anh a 2 fen atte 
POP. (NAME} 
Does Sores Ae te, fk een Fe, Bete = Se a ars 
R CHECK FOR PRIMITIVE 
ens | 


T'H LOOK, FOR J=lylyJeGeNPRIM(O) 
WER NAMESNESNPRIM(J), T'O LOOK. seh oe siete G3 
CODE=PRIM(J).10)} 
ee Se BSN COOR 8 - WSS Ae eee By ts - Bie 22ND, et 
LOOK CONTINUE 
RR 
R USER DEFINED FUNCTION 


T*°H ULOOK, FOR J=1e2sJ3.G.UFUNC (0) 
WER_-UFUNC (J).NE.NAME, T'O ULOOK -- —s = Hoe = 
USTSUFUNC(J+1) 
FAQ _ERR___ 


TO PROC 
ULOGK-- CONTINUE eee ae - 
ERR P*T ERRM, NAME 
—-—-.. -V#S ERRM=$C6,H/ NOT DEFINED. /a$ So sce 23 
FIN =L 


PROC —_____RORaSEQRORILST) 
Loop NEXT=SEQLR.{(RDR, 1) 
ce WR TE&.1 

POP. (NEXT) 
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RETAC. (NEXT) 
FIN STACK{O) -——— - 

O*R NEXT.E-$($ 

Pu SH 04 

NEWTOP. (SPOINT, OPSTCK) 
NEXT=SEQLR. ERDRyT)— — — 22 
NEWTOP. (NEXT,OPSTCK) 

O'R NEXTsEs$)$ ~——— — 
NEXT=POPTOP.(OPSTCK) 


ARGF. (NEXT) 
fi —F*O-KEEP Pee aay - = oe 
Ett 
-—- PUSHS (NEXT) 
KEEP SAVE DATA ROR, BASE 
SAE RE 
BASE=POPTOP.(OPSTCK) 
oe: CODESEVAL.403 a 
RESTORE RETURN 
wae RESTORE DATA BASE, RDR A 
WR CODE.L.0-0R.CODE.E~$FALSES, F'N CODE 


ARGF. (NEXT) 
oe OVE Scie 
PUSH. (CONST(NEXT)} 
— Eft “ oe 
¥'O Loop 
£4 


CONTRL MAD 
EXTERNAL FUNCTION £SPOT} 
NER 
INSERT FILE COMMON 
E'O RETAC. 
STACK ( BASE )=SPOT -- 
SPOINT=BASE 
iN 
E'O PUSH. 
SPOINT=SPOINT#1 
STACK{ SPOINT)=SPOT 
FIN 
E'O pop. 
——- SPUFWSFACKISPOINTY 
SPOINT=SPOINT-1 
FUN 
E'N 


APRIM MAD 
EXTERNAL FUNCTION (DUMMY) 
NTR 

STATEMENT LABEL X 


V'S NRMBIT=4000000000K 
INSERT FILE COMMON 
E'O Le 
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W'R ACHECK.(BSTORE) 
BV=FTEMP2...-FTEMR1——_- eee ne vee 
O'E 
—_______ Be TEMP2.L TEMPE 
Eth 
T!0. BSTORE Bs es silent Be 528 ye as ee es 
E'O LE. 
WIR ACHECK. (BS TORE} ener Ser eat eee = 
BVaFTEMP2.LE.FTEMPL 
eee, ed OE I 
BV=TEMP2.LE.TEMP1 
Eth. eee oid eo oo, os oe es 
T'O BSTORE 
- 10-EQe  — a Ge hg poet asd aebese 
WR ACHECK.(BSTORE) 
——---______B¥=FTEMBILE.FTEMp2 
O'E 
co OVE TEMPLE. TEMP 2 2s <a 
EfL 
—__ —_ 340 BSTORE_-. tetieteas ae bg a Se 
E'O GE. 
——_______WIR_ACHECK.(8STORE} 
BV=FTEMP2.GE.FTEMP1 
Ore. a3 2 Aat 7 Sf aghn eA ae AS 
BV=TEMP2.GE.TEMP1 
nn xo web eece ectenstet pease eee Cee ae eae 
TO BSTORE 
EtO G. 
WR ACHECK.(BSTORE) 
= BVxFTEMP2.G.FIEMPL : ef eee 
O'E 
-- — BV= TEMP2.GeTEMP 1 : eee Jo eet ret 
etl 
asTOREe _RETAC.iBy) 
FIN 
_. £20. PLUS. — 2 wns Sila Eee 
W'R ACHECK.(BACK) 
__ ETEMPL=FTEMP1+F TEMP 2_ eh Set eS 22) 
Ore 
TEMP TEMP Le TEMP? 
E'L 
eee T'O BACK : 2222 
€'O MINUS. 
WIR ACHECK. (BACK)... . : fa ana ee 
FTEMP L=FTEMP2~FTEMPL 
gig. 
TEMP1L=TEMP2-TEMPL 
BAL aes 
T*'O BACK 
- £90 TIMES. 
WR ACHECK.(BACK) 
—_____ FT EMP L=FTEMPIeEFTEMp2 
OfE 
TEMPLSTEMPL*TEMP2 ee ee ee 
EtL 
T'Q BACK ae 
E*O DIVIDE. 
—_____-__WLR_ACHECK. (BACK) 
FTEMP1=FTEMP2/FTEMPI 
Ore 
TEMP L=TEMP2/TEMPL 
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EL 
BACK ~ RETAC.tTEMP LY -- - -~ ~- 
FIN O 


R—__—— 


R 
-———.  - INTERNAL FUNCTION {X)-- -—- 
E'O ACHECK. 
= POPS tTEMPL) 
POP.(TEMP2) 
WE RE MP Be SHENG 06 OR FE MP2 BBE NE 0 § 
TEMPL=$#INC#S 
Kare bd tee BV= 1B ata aie 
To xX 
pas a gta €'t ie scans 
WER TEMP1.AsNRMBIT E20 
—____-_____F IRS ¥=98—____ 
O'E 
= FIRST=18 = 
EL 
—— WER TEMP 2. A.NRMBIT.E.O 
SECOND=0B 
—______91— 
SECOND=1B 
rane E't oasis is Ste ee te 
W'R FIRST. AND. SECOND 
- a FEN 1B 
O'R FIRST.EQV. SECOND 
FiN-OB 
O'R FIRST 
FTEMP 22 FEMP 2 - & 
OE 
-—- FTEMPLsTEMP L 


LPRIM MAD 

EXTERNAL FUNCTION (DUMMY) 
NIR 
INSERT FILE COMMON 
B'N TEMPL,TEMP2,BTEST,BV 
£48 ANB 
W'R .NOTSBTEST.(0), FIN -1 
BV=TEMPL.AND. TEMP2 
T'O STORE 
€°O OR. 
WR .NOT.BTEST.(0), FIN -1t 

EP ES REM PD 
T*'O STORE 
£°O EQV. 
WR .NOT.BTEST.(O), FYN -1 
BV=TEMP1] .EQ¥.FEMP2 
T'G STORE 
£48 NGF 
POP.{(TEMP1) 
BV=.NOT. TEMPE 
T*G STORE 
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STORE RETAC. (BV) 
con —— FAN go — : : peers — 
R 
SO a OR a TES 
INTERNAL FUNCTION [X) 
we EQ -BTEST, = 
TL=STACK (SPOINT) 
wo eee ees os F2RSTACKLSPOINTH=EP- 
WER TLLE.0.OR-TL.ESL 
tS poe temp 
D'R T1.E.SeINCas 
we a POP LTEMPLD ey ee : 
TEMPL=1LB 


NEXT W'R T2.E.0.0R-T2.E21 
we ee ee OP TEMP 2) ae oo cere e — 
O'R T2-E.SeINCs$ 
-—-——--+ - - POP FEMP2)-. - — . --- —— --— — — -- — _ —-- 
TEMP2=18 
—_—_———__ 91 
F°n 08 
ae are 37 pees, Pa eee ea ener a aE Sec aT es ae a Ye 
Fen 18 
Aten ate SEMIN acs ee as Git, Sanus MEseS yee oS: Deh Nee oie wa ce te ee ate aaa a = 
EN 


SYMATS. MAD... -.------- -.--- a Wy hangs Ge figs ee eee on gabe oes Sh 
EXTERNAL FUNCTION (DUMMY) 

i ee 
8'N BV 

oo. cu ENSERT FILE COMMON. $5. acy. a2 eB ere te, ie a eh ee 

E*O PRES. 
POP. LTEMP) —_. . eae ea 
LOC=MEMBER.( TEMP, SYMLST,O) 

ase WER LOC eg a eS 

RETAC. (MARK) 


-OfE ts 
BV=TEMP.G.0 
SYMCNT=SYMCNT+L. eee a Ses chs 
RETAC.{ BV) 
— ao 
Fen 
V*S MARK#$eINCes 
R 
R 


E*O ATTRIB. 
—________Can&=9 
LIST.{LST) 
- -POP.USYMP) |. - 
PoP.{aTT} 
WER. SYMP 2h. 
CODE=$FALSES 
—___________VALaSFALSE$ 
T'O RET 
EL 
LOC=MEMBER.( SYMP» SYMLST+0} 
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CHECKP.{ SYMP) 
28 ec:2FSQINLOGRS ses LSS se SS Saye — 
R 
——R_HERE THE 9 NORMAL! RESULT OF A TEST 15 
R PROCESSED. 


yea te ee eat, Poet St : = : : = 
NCOMP LIST.{SAVE) 
- ++. -PENSCOMP. (TEST /PRIGRySAVE) = — = eee = 
CHECKP.( SNORMS) 
——______IRALST. Ais 
F°N RLST 
= R-.- ea = x sath a Remar jhe debt pals Stesos 
R 
ws INTERNAL FUNCTION (MARK) ee —— 


E*O CHECKP. 
ER RG 
LIST.(SCRAT) 


_ -. _ _—. . MANY. SCRAT,$PROBS¢P, SPRIORS, SAVE, SRESULTS, MARK) — — —_ -- — _ 
NEWBOT .(LIST«(9},RLST) 

=. ==... WAKEDL ASGRATs BOTCRLSTO D2 = ue =: 
IRALST.(SCRAT) 

At ig ee ne es 
IRALST.(SAVE) 

fete ee eee ERE ee het AB oe «yk ee tt eee oi wnee 7 
E'N 

ae _—__€2N- 7 = = Os AG es kee wecles ase te a 


RELTST MAD 
EXTERNAL FUNCTION ILST, PRIOR} — — be Stes es ed oe Ae 
NPR 
______I NSERT_EILE COMMON 
B'N LEMPTY 
_ . _—~ .. STATEMENT LABEL SWITCH 9 -—- ~~ ses Pe eg pe era 
F°T P, THRESH 
Ses EQUIVALENCE (1P,P). ; Sra ee nan 
R 


—-________®_THIS EUNCTION DETERMINES ALL THE TESTS WHICH 
R ARE RELEVANT TO THE CURRENT STATE LIST OF *PRIOR*. 


_-. -___..&@ TESTS WHICH HAVE ALREADY BEEN RUN ARE IGNORED. . _ _ ee 
R 

——.-— -.-EfO- RELIST... ——. eee eases “ghee eee ee 
COUNT=0 


4A B6F4RORS Fi _$§_ 
W'R THRESH.G.1. 


wo ee SWETCHaRET. iS yee are See tess 
STATE=0 
—- ———-. ~ DLOSS~(PREOR, STATE} = fg dth eeeetoas : ees 
T'o DO 
—_—________4£ 
SWITCH=LOOP 
ES, pac et ey, —__ a en — = 2h wet ip P yee seperated b 
SR=SEQRDR.« (PRIOR) 
LOOP... STATESSEQLR.(SR,SH)--. ——. Sy de oe SS = care 
WR STEEL 
F*'N COUNT 
_ ae €tL — i iets Ste 


IP=SEQLR~(SReST) 
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WOR PoLE.THRESH, T*O LOOP 
DOL — ——— -S¥R@SEQRORS (STATED _ —-—— 
INLOOP SYMP2SEQLRe (SYRy SY1) 
Rig Be 
WR sNOToLEMPTY.(RDRSTK) 
ERORSTK} 


T'o INLOOP 
Ore = a 
T*°O SWITCH 
8 eSeSSSSeSSSeSSeSSSSSSSSSSSSSSSSSSSSSSSSSSSsSssesesee 
O'R SYI.L.0 


WSR MEMBER. (TEST)CELL (1991) .NEO. TO INLOOP 

———--~—--- WAR MEMBERS LTEST si STy O}0NE «Oy -F°0 #MLOBR- — — 
WIR MEMBER. (TEST, TSTRUNsO).NE.O, T°O INLOOP 

$$ Ne FF 9 be $$ 
COUNT =COUNT+1 


Topt #aAD 

oo -EXTERNAL_FUNCT ION 4A7EST STATES ———— —----— — 
NPR 

————- $B 
INSERT FILE COMMON 

eee en —FAF_LSAVE pDSAVEYLS— — —~—-—————-— + -— 
EQUIVALENCE (ILSAVEsLSAVE), (ILSeLS) 

ane £40-FOPFH.- ————— = = ———-~ _ 
SWITCH=1 

$$ 4 64 NO6909 
T°O START 

See —€*p- 4OPr.— — - = —- - 
SWITCH=2 

START—_— ——R=SEQROR~ 11 TSVAL~ ESVALUESS, FREED} — —— —— - - 
RET=0 


ILSAVE=SEQLR. (RF) 
Seat —DSAVESLSAVE— —--- ——--- 
ADD=LPNTR.(R) 
LOGP—— ——~-TEST=SEQL Rs Ry F)— — ~ Sh = 
WSR FeEols T°O ENO(SWITCH) 


W°R LS.G.DSAVE 
F190 -SAVELSW ITCH) — sie Stat 
O'R LS«LLSAVE 


LSAVE=LS 
———_—_____AF6 545 F566} eee 
ADD=LPNTR.(R) 
-E4L — ses = 
SAVE(2) T*O Loop 
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SAVE(L) - MEWTOP. (TEST, NOGOOD) 
10 4.008 


ENOCL) WIR NOT. LENPTY.(NOGOOD) 
NEWTOP. (NOGOOD, CELL (1)? 


TRALST.(NOGOOD) 
ENO42}— —— REMOVE 1. NKL LCONT. FADD} 1) --— -—— 
REMOVE. (ADD) 


ee ee ee 


-----— -EQUIVALENCE_(I PROB, 

O°N LEMPTY,MAMTST 

ESE RT_ BILE COMMOM 
E*o SETUP. 

——- —MANYV.ASFRUCT A 1ST ASTL EST) LIST. US YNS) LIST. LIESTISH) —— —__ —— 


NEWBOT o({LIST.(9),STLIST) 


NEWBOT.(LIST.(9), TESTS) 
LMUG _ 


LIST.( TEMP) 
LOOP —-—— -W2R_DSKLST. (41,N2s- TEMP). §. SDONESS.—T10- SNOO — —___ __ _ _----- ---_ — 
WORD=POPTOP. (TEMP) 
——--_______WIR_WGRD Bo SSTATE$—______ 


T°o ST 

—- —O2R_ WORD. E.SSY¥MPS$_—- _— — ES 
T°O SYMPL 

--—— s oE-S$TESTIS$S— _—- _- _- _ —— 
T'O TESTL 


—_——_---__—__ O02 8_NORD..£.SDEEINES.__ 
DEF INE.( TEMP) 


: T°O SPTST 
--~-——— —OLR_ WORD. G.SEXCLUS$ 
T¢O EXL 


PRINT COMMENT SERROR IN BCO TAPES 
Se CMNCON. (0) ee ei ie Se 


Joes. NEMVAL.45PAR0BS OP-(TEMP), STATED = 


—S¥MP=POPTOPW{TENP 
TPROB=POPTOP.1 TEMP?) 


SYMP=LOOKUP. (SYMP, 2) 
Se a Se -AANY A-LTSVAL 1 SHEMBERS » SYMP), STATE, PROB}. = 
NEWBOT. (SYMP, STATE) 


RS a aS ALAND oss 
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T'O STLOOP 


PYT KKy TOPL(TENP)» NTHTOP. (TEMP, 2) 
en VAS -KK=$2 15 ,K12328 =e --————— 
NAME=TOP. (TEMP) 
eee FES Fak COKUP s  HEHFOP of FEMPY 233) — — — — —~——-— 
WR NAMTST.{ MANE) 
5 V0 SOR GAIN y-B}————————— 
NEWBOT. ( SYMP, ITSVAL. ( SHENBERS, TEST)) 
pe 


MEMLST=ITSVAL. (SHEMBERS, TEST) 


RLOOP SYMP=SEQLR. (RSF) 
———_ 44 RFs Ev ks 449 -S¥MPL1—— 7 —— 


NEWBSOT (SYMP MEMLST) 
TESTL WOR LEMPTY.CTEMP), T°O LOOP 


——-+-— AR MAES Fa HABE} ~— _ 
NEWBOT. (COST, LOOKUP. (NAME, 3)) 


—————— RESEQRORT NAME} - - 


NEWBOT .(COST ,LOOKUP. (NEXT, 3)) 


SPTST WER LEMPTY.(TEMP), T°O LOOP 
oe FES Fak GOKUP 6+ POPTOP ol FENP 93) — — — = 
NEWVAL. ( SSPTESTS, $YESS, TEST) 


es -T10-SPTSt— = Gee 

ENDO IRALST.( TEMP) 

—_____—_ fi... 

SYMPL1 POPTOP.( TEMP) 

——-—-— —PopToP. (TEMP) —— — ——— — - —_—-_ -—__ ---—- -_---_-___. 
TO SYMPL 

FESTLA- —— —POPTOPs LEMP} — — ——_— -- + 
PoPToP.( TEMP) 

STEERER ick =A, Sune eREEEen EERE 

EXL W°R LEMPTY.( TEMP), T*O LOOP 


a —TA@ TRANS LPOPT OP LTEMP 113} - - ~ 
T2=TRANS » (POPTOP.( TEMP), 3) 

a NEWVAL = SEXGLUS $y Fi 5T2)- — —— 
NEWVAL .( SEXCLUSS,T2,T1) 

$$ F208 $$$ $$ 
EN 


wa EX TERNAL—FUNGT LON 4 WORD, LCODE }- — 
NER 
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INSERT FILE COMMON 
£10 LOOKUP. 
HLIST=0 

PPUDEN OORT V1. To9 oY) G1 a Fe e  Ee 
W°R ADD.E.0 

aeeneoipen aera NEWBOT(L1ST.(9) ¢HLEST) 
LST=BOT.(HLIST) 

——a WLR _LCODE. Godby NEWVAL (1 SMEMBERSSLIST.(9),LS5T) 
NEWVAL. ( SPNAMES$,WORD,LST) 


F°N LOCATE.(0} 
sg es Se INTERNAL_FUNCT LON 1X) eae 
E'O LOCATE. 


HLIST#NTHTOP. (NTHTOP. ( STRUCT o£ CODE) p HASH. ( WORD, HSHNUM) #1) 
anaes -ROR*SEQROR.~ CHL IST eS es 
Loop LST=SEQLR»(RDRsT) 

a WIR, Bod 8 NOs ee ee 

WER ITSVAL.(SPNAMES,LST).E.WORD, FYN LST 


E'N 
-68N 


INSECT MAD 
—-— -EXTERNAL—FUNCT LON- 4.112513) 
NER 
BL N-LEMPTY 
E*O INSECT. 
ener 
R THIS FUNCTION DETERMINES THE INTERSECTION 
R-OF_L1I_ANO. 12 AND PLACES THE ANSWER IN L3- 


Peas eee RORZSEQROR- (11) = 
LIST.{RORSTK) 
Loop __ ELEMsSEQLR.(poR,ry 
WR IeEol 
WR LEMPTY.{RDRSIK}  —- — = 
IRALST.(RORSTK) 
FON = a eats a5 
O'E 
——-—_-_________ ROR=@PORTOR.(RORS TK 
are 
--- -- --— -- DER Pe bed Aes BS Sid = 
O'R ITSVAL.(SRELAT$, ELEM) .NE.O 
2 A NEWTOP A RDR RDRSTK} — —-- — —__ — = aS 
RDR=SEQRDR. (ELEM) 
——__________D 2 R_MEMBER. {ABS ELEM, L2,0) NEO 
NEWBOT. (ELEM,L3) 
EL =e Spe ee os rey _ 
Ta Loop 
EN. __ ae 


MEMBER MAD 12/12/66 2321.6 92 00000 
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EXTERNAL FUNCTION (GOAL LST, LEVEL) 


NERS 

E'Q MEMBER. 

—________a88=9—___—- 
RDR2LRDROV. (LST) 

DESCND-—— - NAMESADVSWRa(ROR YE} - = Sa “a 
W'R ITeEely T'O RETURN 

So “WER LONTR SA RDR Dok LEVEL y T4G-DESCND = = 

COMPAR WR NAME.E.GOAL, T*Q FOUND 

———___-NAME= ADV tit (R08 
WR TeNEol, T'O COMPAR 

ASEEND- ——-W*R-LENTR. (ROR TSE. Oe 48 RETURN — — <a 
LVLRV1-( ROR) 

-——- ADVLNRs(RDRYH —--——-— --—-——- ~ aaa 

WR I.E«l, TO ASCEND 

48-858 


FOUND ADD=LSPNTR.(RDR) 
RETURN -  - TRARDRsARBRE oo - 
F'N ADD 
ete coe aes pen ee ae Seted — 
UNDO MAD 
——---—- EXTERNAL FUNCTION (LST) a> --———- -- 
NPR 
INSERT EG OMMGN 
E'O UNDO. 
~ <== --— -RDR=SEQRDR «UT TSVAL-«ESSYMPSS)LSTID. - — = ea 
LOOP SYMP=SEQLR.~(RDR; 1) 


-———---WIR-BeEaks F°N——- a re 
RDR1=SEQROR.(PATSTK) 
4.09P4—___ -NEXT=SEQLR ARBRE 
WIR [1.Eel 
— ——-NEWBOT 3 S¥MP)UNACTD} — -— a ------ — 
T*O Loop 
-OLR-MEMBER « (SYMP, ETSVAL~ {SSYMPSS NEXT} O)SNESO  - ———— -- ——- 
T'o Loop 
OrE 
Teg LooPL 
Ef’ ports ee Seattinde, alin tate oe a hae et a al 
E'N 


DSKRDOD MAD 12/01/66 202422 - ~144 600600 OS eee ea mee ee 
EXTERNAL FUNCTION (FIRST » SECOND, ist) 
--NtS INTEGER a= Se a ed 
INSERT FILE COMMON 
NAN NAME FER 2 
E'G OSKLST- 
----. Vt MODE=E --~ aetete sO aes Ses 
T'O START(MODE). 
START EL} ---NAME(0}=RJUST = CFERST) wane re 
NAME(1)=RJUST. (SECOND) 
BF OPENSESRbyNAMELOD NAMED}, BURLL432},BUF2(432} =O, ERR} 
MODE=2 
STARFE2} BFREAD. (NAME(O}sNAME( 1}, ENTEOD wee LeEGFPEOFCT sERRI— — - - —— 
COUNT=LNKR«(INT{OD) 
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BFREAD. (NAME(0) »NAME( 1)» INT(COUNT) eee COUNT, EOF eEQFCT sERR) 
Se kee 4*H SWITCH, FOR L2COUNT,=1,- 1. 6.—0- 
SWITCH OTHER(COUNT-I)=INT(1) 
GT WG RO CU 
K=VLIST.(OTHER,LST) 
(Seas -WIR_K Es -SNOTVETS, 390 START(2) _ 
FIN K 
OF — — — —_BFCLOS 1 NAME(0}, NAMEL 13, ERR} 
MODE=1 


ERR PRINT COMMENT $GOOF ON READING FILES 
: MODE=1 _ 
F°N $SDONES 
EN 


RELEV MAD 

aaa eee EXTERNAL FUNCTION {SYMPR,LST) 
NPR 

—_____ £90 pevey, 
ROR=LRDROV. ( ITSVAL.( SMEMBERS, SYMP} } 

LOOP _TEST=ADVLNR.(RDR, 1) — 
WIR TeEel, F°N O8 

Ses -WLR- MEMBERS ATEST o LST, 0).NE-0, FIN 1B zs 
T*O LooP 


LOSS MAD 
———__SXTSRNAL_BUNCTION (AL, Az} 
NPR 
ers FLT _PReLOSS,WGT, SAVES FCONSWIOT, PI - 
B'N LEMPTY 


fh ee ee ees EQUIVALENCE (IPR,PR),LIWGTWGT) 
D'N PI(10),LOSS(100,AD) 


V°S SUBS=0.05 102202307 405 509604 70,80,90 
eS B24 LempTyY —— pet oe ee 


DSKLST.(Al, AZ. BUFFER) 
ae —SI2E*PORTOP.( BUFFER} = ms 
DSKLST~{ Al, A2, BUFFER) 
a -T2H-LOOR, FOR-J2he-1)J~G-SIZE 
WR LEMPTY.{ BUFFER), T*O ERL 
—_——_-NUM=RORTGR. 4 BURKER J 
NAME=TRANS.} (POPTOP.( SUFFER)» 1) 
ee ree? -W!R_MAME.E.0,_T!0 -ERL = - 


R- 
Loop NEWVAL.( SINDEXS,NUM,NAME) 


T'H LOOP, FOR J#1,1,J.G.SIZE 
pie as ae -IND=SuastJ) = 
W°R DSKLST.(AlsAZe BUFFER) -E.$DONES, T*O ERL 


See ae oe 
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WEIGHT — MAD Pe fay nip ahs 
EXTERNAL FUNCTION (LST) 


F°T WGT se ANSsPR 
ey re EQUIVALENCE (MGT o 1LWGT1,t1PRe2R)— — _ —_—— —_ —_--— 
E*O WEIGHT. 


—--— ANS =ANS*PRONGT — — — — — ——_— -— — —-~— — —— —— — —~— ——— — ———— 


FAST MAD 

_—_—______. EXT SRNAL_BUNCTION_(CONTAL) 
NR 
—BIN_LEMPTY. = ete ee a Fag 
INSERT FILE COMM 


LUST.CTEMP) 
—_—_——___—__PRINT_COMMENT—S¥QU-OR_NES—— 
R'T $C6*$, ANS 
Se WIR_ANS Es $YOUS — at SN A eo er 
CBIT=0 
pene eee eee PRINT_COMMENT—SCONTROL_LISTS— — Pe = 
ROLONL. (TEMP) 
—______beatwapoptop.iremey 
THRESH=POPTOP.{ TEMP) 
les in awh es MENITS=PORTOP..CTEMP) _ es ro aes 
NOISE=POPTOP. (TEMP) 
eae ae ees CONTRL@PORTOP.(IEMP) as Sie = 
PRINT COMMENT SCASESS 
——__—____RDLOML (Teme, 
NUM=POPTOP.(TEMP) 
oe . = foe. 
calt=1 
faye = = ie 


PRINT COMMENT SHISTRY FILES 


WR .NOT.LEMPTY.( TEMP) 

pee ee FALE L2RIUST.(POPTOR. (TEMP) ) oF 
FILE2Z=RIJUST. (POPTOP.( TEMP) } 

a ir aaa liana etait 
E*t. 

RTE T8000 E58 — 
ROLONL.( TEMP) 

——-— CPRIOR=@POPTOP.. (TEMP) — = 
CPAT=POPTOP.{ TEMP} 

eS —ALLPAT 2PORTOP. (7 EMR} _ 
ALLTST=POPTOP.( TEMP) 

——_—_—__—_—_© 74S Tanda ton.t Tena) __ ee 
SIGNS=POPTOP.( TEMP) 

--— ——— -- STAND=PORTOR.{TEMR} - — — -- --- 
WIR CPRIOR.E.2.AND.CBITE.1 


nn Taal 


THRESH=POPTOP.{(TEMP) 


PRINT COMMENT S$DEPTH, THRESH, HEURISTIC 
-—— ——-BEP FHePOPTEP. ({RBLONL = tFEMP) )— < 


CONTROL $ 
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——_________€aNFRE@PaP TAP HEMP 
E'L 

--- -——. IRALST.CTEMPR. ~~ 
F'N NUM 


= €4N is a ae, poet 


FIERMUP MAD 


> a> — — HEX HERNAL—FUNCT EON {PATS 


NPR 


_——_______81N-SUBSEF 
F'T PyPR 

—-- ———---EQUEVALENCE (PR, EPR} —— 
E'O FIRMUP. 

— ae Pee 
R=SEQRDR. (PATSTK) 


PATR=R- 
Loop NEXT=SEQLR.(R+F) 
CHEGK——- -WER-Fafels FAN-P ---— 


CURPAT=ITSVAL. (SSYMPSS,NEXT) 


-——-. --—-RispaTR ———— = 
Loopl CAND=SEQLR.~{R1,11) 
———______#4R-14 £1 


TERC ETSY ALS TSP RDS NERT 


x - P=PaPR 23 
T*o Loop 
— OR CANDSESNEKT 
T'o LooPL 


8! R_SUBSEF et GURP AT HESVAL A SSYMPS$ CANDID 


ADD=LPNTR.(R) 
ee. - NEXT=SEQLRs t{RyF} 
REMOVE. (ADD) 
——-¥*9- CHECK ———-- 
OE 
—_____-__#19-4.9993—_- 


Eftl 
E'Nn 


SUBSET MAD 12/26/66 1718.4 44 


EXTERNAL FUNCTION {(Li,b2} 


NER 
£€°O SUBSET. 
R=SEQROR.~(LI) 


00000 


+60P———_NEXAFOSEGt Rs ty 
W'R FOE] 
-F'N 1B 


O*R MEMBER. (NEXT,»L2:0).E.0 


‘ FON OB 
OrE 
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S¥MS AY —_ MAD 
EXTERNAL FUNCTION (SYMP,TEST) 
oni et NER ee: _e 


INSERT FILE COMMON 
---——— £10 -S¥MSAV. —- 


W'R MEMBER.(SYMPeSYMLST20) .NE~ Or 


FIN 


—_________NEWBOT.{1SYMP,SYMLST) 


UPD. {SYMP) 
FIN - ae ee = 


E'N 
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COMMON FaP 

BUFL COMMON. —433- 

BUF2 COMMON 433 

cBit Common} 
SIGNS COMMON 1 

GPAT— — CGMMGN-—1 

ALLPAT COMMON 1 

CPRLOR- COMMON —1- 

CTEST COMMON 1 

ALALTS T_COMMON he 
DISEAS COMMON 1 

STAND — COMMON —1 

FILE COMMON 1 

FILE2 — COMMON —1 = 

DEPTH COMMON 1 

THRESH COMMON) 
NINITS COMMON 1 

NOISE— COMMON —12- 

NODES COMMON 1 

BASE -— COMMON —1- _ —_ 

TREE COMMON 1 

CASAS FCO 
PATLST COMMON I 

STRUCT-COMMON —1 

SYMCNT COMMON 1 

S¥MLST-COMMON-—1 

UNACTD COMMON 1 

FSTRUN COMMON Ye 
PATSTK COMMON 1 

CODE -- COMMON —1- - _ — 

OPSTCK COMMON 1 

STACK — COMMON —21 = ee 

UFUNC COMMON 21 

ARGS ___COMMON 2} 
PRIM COMMON 31 

CONST— COMMON — 31 eee. 

SPOINT COMMON 1 

NPREM— COMMON — 31 Ze 

CELL COMMON $21 


MAGROS-— FAP 2 SP ce Ss he ees at = 


PUSH MACRO ARGS 


ire ARGS —— SoS 

TX1 etlelel 

CLA ARGS — -— coe _ ~ -- — 
sto STACK, 1 


ee Se ee rc eee eye ec ee ne a 
PUSH END 


- agi tn See 
pop MACRO ARGS 
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IRP ARGS 


CLA - STACK st - -—- —--~—- 
STO ARGS 
TH e+trivt 
IRP 
pop. €ND --- ae a ————- 


* 


* LISE- READING MAGROS HERE -- Pease Z es See ene 


SEQGADR-MAGRO— Ay 

CLAe oA GET LIST HEADER 
eS sto 8 a STORE -IN-READER-CELL pts 
SEQROR END 


* Z 
SEQLR MACRO A,8,C 
————$_ + Ae _-——_ 89 4__———_—\___~—_ READER + atl 


CLA 154 GET DATUM FOR CELL 

-— $18 — ie a - —— SAVE DATUM < : Senn anne 
CLA 014 ADVANCE READER 

-—S§T9- 8 ae seas x . eee 
ANA =0700000 SET FLAG 

+ AR SS 3 
SuB =1 
--— §TQ --—€ = ae sls ee a es weggee Se CBO 


SEQLR END 


ENTRY NAMTST 


NAMTST SXA& SV¥4,4 ——- : -— =e 
CLA® 14% 
SFO CAND 
TSX SGETMEM,4 
— — FH ~ = — - - -— 
sto LIMIT 
— CLA ---GAND = - ee -——-- 
SsP 
Se -HENK 
ARS 18 
CAS -- LINK. — - os 
TRA NO 
FRA - a2 == ‘ - 
TRA NO 
——Ett HNIC 
CAS LIMIT 
--— FRA NO = ---- 
TRA eel 
= ClAe ~ LENK - 
STO HEAD 
———~-ANA—§|_-—=8700600- 
CAS #0200000 
FRA NO _ a 
TRA #+2 
FRA NO - - - ad 
CLA HEAD 
ARS +8 
CAS LIMIT 
FRA NO = ec 
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STA o+] 
GLA oe =e Sai es, _ 7 — aa _ 
ANA =O7777T7 
GAS ——__4 kK ——- 
TRA NO 
TRA -YES- : tae See eS Settee = 
NO CLA =] 
FRA — .- 42 — -. a Patten Yacy tS ee aE my koe Peano 
YES CLA =0 
$V4—__-AXT____aa,@ 
TRA 204 
GAND.. -PZ€ -— -—- — oo = Se - 
HEAD PZE 
LINK — PZB —— — -. -—-— »- - —-— ee a oe — — 
LIMIT PZE 
a ee 
SLF FAP 7 7 
# *GROWL® IS USED TO GROW NEW BRANCHES ON THE TREE IF 
@ NECESSARY. —— - os -- = oe —-—-—— ——— 
* 
B= —_ a Sam tare! 7 Se 


® STACK MANAGEMENT MACROS 


IRP ARGS 
TX e+l, lel . sec eras ee ee 
CLA ARGS 
5S 5. A$ 79° S_ STACK A = foo, RF Eat = = 
ERP 
RUSH: ENO = 
* 
POP _ MACRO-.__ARGS eee ae = 
IRP ARGS 
CLA STACK, 1 nO 2:8 Bt “i 
sto ARGS 
ee Es ee Se ns 
IRP 
pop. END. ; = ei Bie oer 
* 
# LIST READING MACROS HERE == Mae fate Ns 
* 
SEQRDOR-MAGRG——Ay BN 
CLA® A GET LIST HEADER 
———— S$TO—-. -B ——_.. | -. . -STORE IN-READER-CELL.. 00. 
SEQROR END 
2... ee mttaer, Wester tle ee Be ee Bint ts arses a = 
SEQLR MACRO A,B,C 
Link 
CLA 1y4 GET DATUM FOR CELL 
wn SEO A _.-_ SAVE _DATUM agate ete Met Ae 258 Be 
CLA 04 ADVANCE READER 
—.—— $70 8 oapee : ft So auees pphe oo ee 
ANA 20700000 SET FLAG 
ARS 15 eee eee 5 eee 
SUB =1 
: sTo ct ie ediee: 
SEQLR END 
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ENTRY SEQDEC 
REF y+ 


SEQBEC SKA 

SKA RET¢1,2 

SKA RET+254-- eS ae 
* INDEX REGISTER 1 IS THE “POINTER TO THE Top OF THE STACK 

LXA ZEROyt --——-- -—- ae 


® INDEX REGISTER 2 IS THE LEVEL COUNTER FOR THE SEARCH 


CLA®e 14 
Hs STB List ee ae = Spe she sean os 
CLA® 394 
-— $¥9—- STATE ee SSS yo Se see RSS 
CLA DEPTH 
——_—$ 08 9 

ALS 18 

STO LTESt. . sa = as pare en = 
TSX STTSVALs4 

-¥XH- VALUE - VALUE-LIST FOR TOP LEVEE — — ————- 

TXH LIST 


——____$19-—_vaLues — 


* 
* THIS 1S THE MAIN SEARGH LOOP. —--- a ~ = - --- 


® FERST GET FHE DECISION LOSS-OF_THE CURRENT PRIGR —-— —— --— 


Loop) CLA NODES COUNT DECISION NODES 
ADD. =t 
sto NODES 
---¥$& ~~ $TTSVALy4 GET—DISTRIBUTEON FOR..THES—NODE- 
TXH PRIOR 
— TMH LEST ——- = sale See 
STO PLIST SAVE NAME OF PRIOR LIST 
tt Fa — 
STO DECIDE 
NOFERM TSX $OLOSS;4——- OECESION LOSS-FOR DISTRIBUTION ——— --———— 
TXH PLIST 
-¥XH  -—DECIDE ~ - DEC EDE-NAME -—-- er ee er 
STO LSAVE 
TSX SMANY.4 SAVE DECISION VALUES IF AT LEVEL ZERO 
FXKH VALUES — eS Aa 
TXH DECIDE 
~~ EMH LSAVE —- -- 
LTEST XL DOWN, 2,6 CHECK LEVEL AGAINST DEPTH 
8 NRE 


J 

* HERE THE LEVEL IS-LESS- THAN THE REQUIRED DEPTH. spe es ae 

* THE TREE IS DEVELOPED TO THE NEXT LEVEL AND THE SEARCH 

* CONTINUES» ee 


DOWN 45% _—$RELT 5 F4 GE F_RELEVANT_FEST5 FOR -FHES LEVEL 
TXH LIST 
TH - PLIST aie 


* 
© PROCESS THE BRANCHES AWAY FROM THE NODE DENOTED -BY—‘LEST*« 
#* EACH BRANCH CORRESPONDS TO A DIFFERNT TESTING ALTERNATIVE 
* ATF FHE -NODE-BENGFED-—8¥ $415 74, 


SEQROR LISFsROR ESTABLISH READER-FOR LIST png aT a 
SEQLR TEST,ROR,I GET NEXT TEST 
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READ CLA I 


cas ONE —. = 
TRA NOHE AD NOT A HEADER 

——...- FRA ——__ 86.2 $$ __ 
TRA NOME AD 
TXH 2925240 = = oa = 24 
TRA RET END 


— —— — FX1- —— CONTIN: 2)-1-— NOT_THE_ END OF THE—-ANALYSES- _ 


NOHEAD-CLA ——-—ZERQ——-— 7 an 
sto ELOSS EXPECTED LOSS FOR THIS TEST 

_—_—_— TSX _$6ROW1,4 — — __*GROW! RESULT LIST FOR. THES TEST. ———--—__ 
TXH ROR 
$TO RESLST NAME OF RESULTS LIST 

*_ SAVE- VAR FABLES HERE Ee = 
PUSH {RDR,LSAVE,RESLST,PLIST) 

——— _ 38% — —_SNEWTOP,4—— — PUT_THIS TEST ON TEST STACK Bs ee 
TXH TEST 


$a —_—___ 4S a $e 


* 
© PROCESS ALL POSSIBLE RESULTS FORTHE TEST CURRENTLY SEING-— ——— ___ _— 


@ EVALUATED. 
- rae 


SEQROR RESLST.RORL READER FOR RESULTS List. 


CLA n CHECK FOR HEADER 

Cas ONE te Es 

TRA GOON 

FRA-———0#2----.. HEADER — Ee te se 
TRA GOON 


, Ne ———ee 
# ALL RESULTS FOR THIS TEST PROCESSED. RESTORE VARIABLES FOR TEST 
# EVALUATION : : eae eee are ae nl fe 


_- —— — POP. PLIST» RESLST/LSAVE,ROR}D.. on 
TSX $POPTOP»4 GET THE TEST NAME 
PsN eS rmiggg a 
sto TEST 
he §K A §80T 58. = SGET-TEST COST: ee et 
TXH TEST 
Bass = FAD = €L08$— 2 =) COMBINE WITH ELOSS.— .. -— —--_ pe Sree OO 
STO ELOSS 
TSX SMANY 5 & SAVE VALUES IF LEVEL IS ZERO 
KH -- VALUES. —- : ee ala. Bute witteciee Sieee eh 2 
TXH TEST 
2 pe PRNSS SLOSS Se : Je) daplge = dyer = hie soehadi 
CHECK CLA ELOSS 
———_ 4 $8. SA¥G—_IS THIS TH6-S5S1 IO DATE — 
TPL DEL NO 
w--- CLA. ELOSS—-—._ .. __. BEST. SO. FAR ge ta RE ee chon 
STO LSAVE 
DEL —- SEQLR— —TEST, ROR, — REMOVE THIS BRANCH bi ear oe ctaa tyes he ek 
LXO ROR, 4 
EG a ga i a a 
TSX SREMOVE» 4 
XH TEMP. ; es oe ae ARS 


TRA READ 
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* 

“© PROEESS—A-SENGLE TEST RESULT HERE --— ——-- 
* 


GOON PUSH (ROR1L, ELOSS»,LIST) 


— Fe --—400Py 2) 4 —-- — even — — ——-.~— So 

* 

“© FOLD: FHES- BRANCH BACK -IN- TERMS—OF -EXPEC-TED- VALUE - in 

* 

CON EEN- POPE EST EL OS Sy RBREF-RES FORE WARE ABLE § 
TSX SITSVAL 24 

-S FXH- — ——PROBO —— ----— GET -PROBAB IL FY—GF -THE- BRANEH:- ——— —— ———- 
TXH LIST 

aa St0———pROB—-— —— —--—--—-—-— - ——-— +--+ 
LDQ LSAVE EXPECTED LOSS 

————#p———pn gg 
FAD ELOSS 

a STO— —~—-ELOSS————--. a--- = —~——-———- 
TRA READ1 

- = Ee. Fe ae ee a pe 

: 

RET a$$ pp 
AXT ee,2 

- ANE one — -- a — 

TRA 1,4 

pear : _ nee _ 

e 


-PATOR —BC-E Ly PREOR ~--- - -— 
TEMP 

-FES7.- — eaeureeten — cae aan ae See ae _ 
ROR 


-FL- ee ca ta, 5 ye mS a ie A kita ile hinds ar ee er ahaa _ 
LIST 

ELOSS— - --—----- - on -a 
ONE OCT 1 


RES SF 
LSAVE 


ZERO OCT ~~ — 9 --— eat oe - aa 
PROB 
PROBQ— -BCI-~-.--2,PROB ——---- Sree penta aes 
STATE 
“BEE 
VALUEQ BCI 1. VALUES 
VALUES ee egeets vs == eee 
INSERT COMMON COMMON PACKAGE 
te END recite Se ds eae am Me > tag ere phate men a Beh feed = _ 
UPD1 ~—s-FAP 


BOSE. a GaGa dnp Seated ternas 
® THIS FUNCTION UPDATES THE PRIOR OISTRIBUTION IN 
* IS STORED IN *LST2*. 


ENTRY UPD 


INSERT 


MACROS 
SUBST — MACRO -READER, DATUM —- i - ae a 
LAC READER+4% 
GLA O44 
ARS 18 
PAC. 044 ae sae _ 
CLA DATUM 
sto 1le4 
SUBST END 
UROL SX¥A _ RET,@ 
CLA* 1.4 
—_—$10 - S¥MP. tee = as S 
CLA 204 
-~$T90 sti SS: Gee cectEare 
CLA® 394 
$4.0 $F 
CAS LST1 
-- FRA OIF = = z 
TRA SAME 
OIF STZ SWITCH - ce 5 
TRA #42 
SAME STL SW TCH 
CLA FZERO 
STO Pp ——— = os 
TSX STTSVAL 54 
— TXH— ._ _MENQ _ === 
TXH SYMP 
$to-- MEME ST 
SEQROR LST1,RDR 
LOOP... SEQLR-. STATE,ROR,T _—- —_ 
CHECK CLA I 
CAS — -—ONE oe a _ 
TRA MORE 
FRA 2e2 
TRA MORE 
CLA. —..-P.— a - -S 
FSB FZERO 
= TPL -NOZERO faa oe 
CLA P 
RET AXL RET,4 
TRA 414 
s s 
NOZERO SEQRDR LST2,RDR 
AGAIN SEQLR- --STATE,ROR,I Z 
CLA I 
LAS NE 
TRA #2 
TRA REF-1 
SEQLR PR»RDORGI 
CLA PR 
FDP Pp 
———__- $30 PROB 
SUBST RDR,PROB 
TRA AGAIN 
* 
MORE SEQLR PROB,RDR,I 
TSX SMEMBERY4 
TXH STATE 
TXH MEMLST 
TXH ZERO 
TNZ ath 


238 


CLA 


FZERO 
ste PR SSeS = = 
TRA STEST 
PURE By 
CLA 04 
PAC O94--- -- - ——— SSS 
CLA 194 
 -— $F PR SSF = 
TSX S$PISe4 
I — 
TXH PR 
sto PR: - Ss = 
STEST CLA SYMP 
—— FPL MULT SSS aad 
CLA =1.€0 
SB IR 
sT¥o PR 
MUL F-- —£0Q—- --— PROB perio = 
FMP PR 
-$T0 PROB- tee - 
FSB FZERO 
a 
TRA SCRAP 
- €LA P- = = 
FAD PROB 
sta Pp - Se ar = 
zeT SWITCH 
—— HA 
TRA DIFPRO 
SUBST ROR, PROB — -- -- —-—-—-— 
TRA toap 
e+ ee 
DOEFPRO TSX SMANY ¢ 4 
FXH 4SF2 
TXH STATE 
FXH PROB SSS 
TRA Loop 
8 4 2 
* 
SERAP—NZF $tHGH 
TRA Looe 
Lac RDR4 = 
CLA Or4 
moe = — ARS - 18- -- = a 
STA ADD 
Ett RGR 
ARS 18 
STA ADB1 mae 
SEQLR STATE,RDR, 1 
FSX SREMOVE D4 
TXH ADD 
F5% SREMOVE 14 
TXH ADDL 
TRA CHECK 
* 
* 
FZERO OCT 233000000000 
-ONE OCF + 
ZERO oct 0 
SWI FEH SUR aes Bet 
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STATE 
EL 
usTl 
$F 
SYMP 


MEMQ—— SCI -___ _1, MEMBER. 
AnD 
AODL _ 
Pp 
cS anne Senn Senna 
PROB 
MEML ST — — we ee 
END 
MEMBER. FAP_-.-.. --- Ke eo 
ENTRY MEMBER 
—-—-— — IWSERT —mMACROS-— — — — — — —— —— —— — — — — — —— — — — — — — 
MEMBER SXA RET» 4 
——__—__SXA—____RET21,) ee 
CLA® 104 
— $10 GOAL— - - _- 
Clae 24 
Sto List. 
CLAe 394 
—___ Fal 6464.4 
CLA List 
$7o NEXT 
TSX ONCE, 1 
RET AXT 00 yh sa 
AXT ee, 


-_ pa 

LEVELL SEQROR LIST,ROR 

LOoP__ SEQLAR.._NEXT,ROR,I — = 
CLA 1 

aS engi 
TRA GOON 
IRA __2a2 ws Spee a ea te ta : : 
TRA GOON 
Zac = es 
TRA RET 

GOON——¥5X ONC - P———— 
TRA Loop 

oe a _ = =) Aa, ne —~ = eT 

e 

ONGE—— SEQRDR-—_NEXT.,R-—— — —____ - Ae = = - 


OLOOP SEQLR CAND, RF 
ELA 


CAS ONE 
ee = FRA A MORES. 5 es Beets Sent Set Br Se 36 Sto oe 
TRA #2 
FRA -_MORE -- a a 
zac 
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TRA #42 
FRA eteap -- - 
LAC Ry ; 

$5 $98 
ARS 18 

-- = ANA— 2077777 ee = Behe 

TRA RET 

‘a ae a = Sed cane 


--E— ites fa a pa pt pp en ar i a a a a fat a a ig Neg oh a si lls a as 
R 
= eis ee en AD A he eg nee Se SE Ne NS Se ee 
GOAL 
EES eee 
ONE oct 1 
- - — END = a eae sade a ee a a a a es ee 

UN FAP 
+ oo — ENTRY —UPBE ~ Hoe ae 

ENTRY NSCOMP 

——— — INSERT — MAGROS- ———-- — 


SUBST MACRO READER, DATUM 
READER esses 


CLA 04 
ARS -- 13—— oat ure yeas 
PAC Oe4 
ELA BATU — a =4 - 
sto 104 
4085 t—_ENd———$$ $$ 
* 
HCHECK- MACRO. —LABLy LAB2, FLAG— co 
CLA FLAG 
---—— GAS —— —ONE-—-— ay _ Chie = as 
TRA LABL 
a BO 
TRA LABL 
HCHECK END~—- peace See - - ee 
e 
#* *UPD14 DGES -THE-STANDARD UPDATE -OF £ST1 INTO £S7T2.— — = 


#* WHEN A SYMPTOM IS THE *AGENT*. 


~_-—_— > 


uPDL STI INDIC 

So RER- 0-3 -— —— —— ~ -SAVE-AND- SET INDICATORS on 
TRA START 

7 Be feees RS a at Reamer? Site & ae 

* "NSCOMP® DOES THE NORMAL UPDATE WITH A TEST AS THE AGENT. 

Pe 

NSCOMP STI INDIC 

Sor RER eS ET ied eSecSe e B eee Se 

SIR 1 

START - SRA—---—-REFY4 me eH 
SXA RET#+1el 

6b a4 4 2G ee 
STO AGENT 
€LA® 294 > ~~ - FIRST EIST — ——— —--—- ---- ~~ oo 


sto tsTl 
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CLA® 324 SECOND LIST 
ate $10 -— tST2 Soatenentadiaicenticieteetieatedinaiiatiataienteaetaendiamtentiaamtatemtan 
CAS LsTl SAME LIST.Q. 
i 
SIR 2 YES 
CLA —EERD. — oe 
sTo P 
-— —-— — F§% — ——_$H FSVAL + -- --—-+ + — 
TXH MEMQ 
——_}tH____AGENT—______GET MEMBER LIST OF AGENT 
STO MEMLST 
- a —— ded aad pang tase Saat Wess mh Le scoala lar trees, Pen sl rd a eens ee iene 
# PROCESS EACH STATE ON LST1. 
oe —.  -. some i Soaps <n = wet para es a 
SEQRDR LST1,ROR 
CHECK HCHECK MORE,NORM,I 
oo ie fae, an sent < At a ins pa ta ein fala Yinoh) ian! yak Loi ae See 
CLA P 
RET AXE O64 Set. cs Sik MELE Seat Bin ig Seinen a eat bag Paes 
AXT one] 
——$_!_$§_.01<$_ ol $$ oe eee 
TRA 494 
eo. e Searing Rien Bt Fs ete tapes tet ater, Set Wp ieee eee 


* 

NORM — SEQRDR —LST2,RDR ——— NORMALHUE 4ST2 -— -—— —_. __.-- --. ---- - - 
AGAIN SEQLR STATE,RORsI 
———__HCHECK-—DIV,RETel,p 
DIV SEQLR  PR,RORyI 


ae CLA PR aa fe Nes ein oe pe ae See = 
FDP Pp 
—— §1Q— -——PROB- —? ee Beate A soho te Ree, hee 


SUBST RDR, PROB 
——____TRA__AGAIN— 


* 

MORE — SEQLR___ PROB, ROR- I — ee ee eee 
RFT 1 TEST PROCESS SWITCH 

TRA NC NSCOMPY ee ees 
TSX GETP,1 GET P(AGENT/STATE) 

MEM Gf 
TXH PR 

~—— ChA-—--. AGENT... CHECK. FOR NEGATIVE RESULT. 0 2 

TPL MULT 

Ene = CUA 2 BO Se : beset 2s 
FSB PR 

———__$¥9 —___ pa 

MULT CLA =1.E-6 CHECK FOR "ZERO* PROB 

FSB — PR : 

TMI OK 

RIEST RFT 2 bo phe tty Be Oe 
TRA SCRAP 

—_____ TRA—__Loop 

OK toa PROB 

wa EMP aes PR eee se . atta cars s oe PO APTS 
STO PROB 
PIGGA <2 tee2 BS 3a eck 

FAD PROB ACCUMULATE PROBABILITY 

—____s19___-p 
RFT 2 AGAIN TEST LISTS 
TRA SAME 


TSX SMANY»4 
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TXH LsT2 
FH STATE : pateraeas 
TRH PROB 


SAME SUBST ROR, PROB 


— Loop ~~ - 

* 

NC——— ELA a1 s£0 JRL TEAL TZE PR ---- 
sto PR 

NLOOP SEQLR AGENT,» Ro F NEXT SIGN 

— —— — HEHEEK- —GBONy MUL FF -— 

GOON TSK SITSVAL +4 MEMBER LIST OF SYMP 

—FRH MENQ—— —— — —————-— —---—~——-— - 

TXH AGENT 

$5 9 ——$§ 6 86 {$$ 
TSX GETPs1 

———— FeH— ——S¥MEM— —— —~ — ~ - 
TXH TEMP 
cia PR oe 
FSB TEMP 

————$ 9 pa 
FSB =1.E-6 TEST FOR ZERO 
FPL NLOGP ——— ~ 
TRA RTEST 

- i as Rah ten 


# GET THE PROBABILITY OF A SIGN GIVEN A SYMP 
Dee e a ee ee eee ee eee ee ren ee 
GETP CLA lel 

$10 HOLD— See 


TSX SMEMBER» 4 
Tat STATE ies oat 
TXH HOLD 

——_FxH 26 9 
TNZ 046 FOUND 

— Sta ZERO- —--—-— 

BACK STO® 2,1 STORE RESULT 

- FRA ay a 
PAC Or4 GET PROB CELL 

$0 A$ pf 
Pac Or4 

——~~— 6tA-—— 4,4 — — -——— ~—-— +. 
sto HOLD 

————-STA REGHT eee a 
ARS 18 FAST CHECK FOR NAME 

0 
CAS RIGHT 

~—— FRA NONAM — NOT —A-NAME-— --——--— — - — - -- - 

TRA o+2 

-- =- FRA — NON a ee vee 
TSX $PIJe4 POSSIBLY A NAME 
TXH HOLD 

—-—-—— FRA BACK ae ae ee 

NONAM CLA HOLD 
FRA— ——-BACK — ee Spain sendin Same 

* 

SGRAP—LXD-——— ADR ye 
SXA ADD 4 


——— — SEQER- — STATE ROR, F —--- --— - — - - ee 
LXD ROR» 4 


Se, ee ee tc 
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SXA ADD1,.4 


TRA CHECK pepe = 


ZERO- — OCT o- -_ ee 
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